2024-09-21T05:25:44.057 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T05:25:44.085 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T05:25:44.170 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913496' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: 1843221895adccfb3b692060251dd6a0183bc410 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/SNdKPaLVjldUyq9yTy5znZrWtnj3nBC8avUGO3UWRezbr4gbIvuTiqHVjOGlt1qnWAJ+uW+Ofypfsn3Qb+v4= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBIVwbK+4pvG4fdwEUs7BVVVpsL0Me72ATzFRHUq83+i1reDE8iwBnibOwJjlJ68DhcmJqamlnn4MEZXwdo+YLc= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBGAVi6K6R0Lz6i1FLH7D6QSAmu3DCdQrChkpvlxI7V4UQKVKDLmOJPz0kShdkTC1MOyrD8Dy3oGIOVqKusDQM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T05:25:44.170 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T05:25:44.171 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T05:25:44.171 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T05:25:44.172 INFO:teuthology.task.internal:Checking packages... 2024-09-21T05:25:44.192 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T05:25:44.192 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T05:25:44.192 INFO:teuthology.packaging:ref: None 2024-09-21T05:25:44.192 INFO:teuthology.packaging:tag: None 2024-09-21T05:25:44.192 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T05:25:44.192 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:25:44.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T05:25:44.337 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T05:25:44.338 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T05:25:44.344 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T05:25:44.344 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T05:25:44.366 INFO:teuthology.task.internal:Saving configuration 2024-09-21T05:25:44.387 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T05:25:44.393 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T05:25:44.415 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 05:20:44.652186', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/SNdKPaLVjldUyq9yTy5znZrWtnj3nBC8avUGO3UWRezbr4gbIvuTiqHVjOGlt1qnWAJ+uW+Ofypfsn3Qb+v4='} 2024-09-21T05:25:44.434 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 05:20:44.655559', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBIVwbK+4pvG4fdwEUs7BVVVpsL0Me72ATzFRHUq83+i1reDE8iwBnibOwJjlJ68DhcmJqamlnn4MEZXwdo+YLc='} 2024-09-21T05:25:44.455 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 05:20:44.654108', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBGAVi6K6R0Lz6i1FLH7D6QSAmu3DCdQrChkpvlxI7V4UQKVKDLmOJPz0kShdkTC1MOyrD8Dy3oGIOVqKusDQM='} 2024-09-21T05:25:44.456 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T05:25:44.462 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-21T05:25:44.462 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-21T05:25:44.462 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-21T05:25:44.462 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T05:25:44.552 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fed15f7c160>, signals=[15]) 2024-09-21T05:25:44.552 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T05:25:44.558 INFO:teuthology.task.internal:Opening connections... 2024-09-21T05:25:44.558 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:25:44.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:25:44.632 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T05:25:44.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:25:44.705 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T05:25:44.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:25:44.779 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T05:25:44.785 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-21T05:25:44.803 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-21T05:25:44.804 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T05:25:44.861 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T05:25:44.862 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-21T05:25:44.879 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-21T05:25:44.897 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-21T05:25:44.897 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T05:25:44.956 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T05:25:44.957 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-21T05:25:44.979 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-21T05:25:44.997 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-21T05:25:44.997 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-21T05:25:45.055 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T05:25:45.056 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T05:25:45.057 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-21T05:25:45.079 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T05:25:45.096 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T05:25:45.126 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T05:25:45.126 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-21T05:25:45.129 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-21T05:25:45.131 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-21T05:25:45.148 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T05:25:45.159 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T05:25:45.159 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-21T05:25:45.186 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-21T05:25:45.189 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-21T05:25:45.234 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T05:25:45.309 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T05:25:45.323 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-21T05:25:45.323 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-21T05:25:45.323 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-21T05:25:45.323 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T05:25:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:25:45.340 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-21T05:25:45.398 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-21T05:25:45.398 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-503.el9.x86_64 2024-09-21T05:25:45.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-09-21T05:25:52.017 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.3 MB 00:04 2024-09-21T05:26:04.109 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 20 MB 00:10 2024-09-21T05:26:08.793 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.5 MB 00:00 2024-09-21T05:26:11.085 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:01 2024-09-21T05:26:11.906 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-21T05:26:17.512 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-09-21T05:26:19.435 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T05:26:19.435 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T05:26:19.540 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-21T05:26:19.545 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-21T05:26:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-21T05:26:19.545 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-21T05:26:19.545 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-21T05:26:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-09-21T05:26:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:26:19.547 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 79 M 2024-09-21T05:26:19.547 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 126 M 2024-09-21T05:26:19.547 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-21T05:26:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 65 kB/s | 36 kB 00:00 2024-09-21T05:26:21.080 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-21T05:26:21.772 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-21T05:26:22.281 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-21T05:26:22.281 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-21T05:26:22.281 INFO:teuthology.orchestra.run.smithi028.stdout:Total 29 MB/s | 79 MB 00:02 2024-09-21T05:26:22.993 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-21T05:26:23.385 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-21T05:26:23.385 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-21T05:26:24.373 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-21T05:26:24.373 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-21T05:26:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-21T05:26:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T05:26:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T05:26:30.426 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T05:26:30.550 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T05:26:35.676 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T05:26:35.718 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:27:20.058 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:27:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:27:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:27:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T05:27:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T05:27:20.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T05:27:21.342 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T05:27:21.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:27:21.342 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-21T05:27:21.342 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.343 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.343 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.343 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.343 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:27:21.343 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-21T05:27:21.675 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T05:27:21.760 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.761 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T05:27:21.761 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T05:27:21.761 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T05:27:21.761 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T05:27:21.762 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-21T05:27:21.762 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T05:27:21.762 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T05:27:21.819 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T05:27:21.820 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T05:27:21.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-21T05:27:21.862 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T05:27:21.863 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T05:27:21.864 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-21T05:27:21.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:27:21.864 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T05:27:21.864 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T05:27:21.864 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-508.el9.x86_64 2024-09-21T05:27:21.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T05:27:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-21T05:27:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T05:27:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-21T05:27:23.923 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-21T05:27:23.938 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.1BP1bvr50o 2024-09-21T05:27:23.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1BP1bvr50o 2024-09-21T05:27:24.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.1BP1bvr50o 2024-09-21T05:27:24.143 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.1BP1bvr50o is 6KB 2024-09-21T05:27:24.191 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.1BP1bvr50o 2024-09-21T05:27:24.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T05:27:24.275 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T05:27:24.275 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T05:27:24.275 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T05:27:24.276 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T05:27:24.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T05:27:24.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-21T05:27:54.440 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-21T05:27:54.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-21T05:27:54.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:28:54.443 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T05:29:03.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-21T05:29:03.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:29:03.756 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-21T05:29:04.121 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-21T05:29:04.121 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-508.el9.x86_64"... 2024-09-21T05:29:04.121 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-21T05:29:04.138 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-508.el9.x86_64 2024-09-21T05:29:04.138 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T05:29:04.138 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T05:29:04.138 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T05:29:05.138 DEBUG:teuthology.parallel:result is None 2024-09-21T05:29:05.139 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T05:29:05.154 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T05:29:05.155 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T05:29:05.157 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T05:29:05.160 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T05:29:05.178 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T05:29:05.209 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T05:29:05.240 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T05:29:05.240 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T05:29:05.243 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T05:29:05.245 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T05:29:05.533 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T05:29:05.544 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T05:29:05.544 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T05:29:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:29:05.561 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T05:29:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:29:05.579 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T05:29:05.581 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T05:29:05.604 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T05:29:05.647 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.660 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.678 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.679 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.683 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.698 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T05:29:05.700 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T05:29:05.710 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T05:29:05.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T05:29:05.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T05:29:05.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T05:29:05.743 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T05:29:05.755 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T05:29:05.755 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T05:29:05.785 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T05:29:05.787 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T05:29:05.803 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:05.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:05.936 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.065 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:29:06.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T05:29:06.131 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:06.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:06.227 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.354 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:29:06.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T05:29:06.421 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:06.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T05:29:06.533 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T05:29:06.618 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:29:06.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T05:29:06.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-21T05:29:06.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-21T05:29:06.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-21T05:29:06.726 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T05:29:06.733 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T05:29:06.761 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T05:29:07.253 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T05:29:07.261 INFO:teuthology.task.internal:Starting timer... 2024-09-21T05:29:07.261 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T05:29:07.290 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T05:29:07.337 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T05:29:07.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-21T05:29:07.402 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-21T05:29:07.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-09-21T05:29:07.469 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-09-21T05:29:07.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-21T05:29:07.525 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-21T05:29:07.527 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T05:29:07.527 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-21T05:29:07.562 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-21T05:29:07.563 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-09-21T05:29:07.612 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-09-21T05:29:07.612 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-21T05:29:07.635 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-21T05:29:07.635 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-21T05:29:07.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T05:29:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:29:07.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T05:29:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:29:07.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T05:29:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:29:07.722 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T05:29:07.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-21T05:29:07.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-09-21T05:29:07.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-21T05:29:07.826 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T05:29:07.835 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T05:29:07.845 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T05:29:07.846 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi129.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T05:33:15.851 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-21T05:33:15.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-21T05:33:15.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:33:15.931 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-21T05:33:16.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-21T05:33:16.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T05:33:16.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:33:16.087 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-21T05:33:16.174 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T05:33:16.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T05:33:16.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T05:33:16.260 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-21T05:33:16.338 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T05:33:16.339 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T05:33:16.352 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T05:33:16.353 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T05:33:16.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T05:33:16.356 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T05:33:16.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T05:33:16.358 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T05:33:16.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T05:33:16.386 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T05:33:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T05:33:16.401 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T05:33:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T05:33:16.409 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T05:33:16.423 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T05:33:16.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-09-21T05:33:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-21T05:33:16.437 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-09-21T05:33:16.445 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-21T05:33:16.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-21T05:33:16.451 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T05:33:16.459 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-21T05:33:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T05:33:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T05:33:16.473 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T05:33:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T05:33:16.487 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T05:33:16.522 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T05:33:16.525 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T05:33:16.526 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T05:33:16.526 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.526 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.526 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.526 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.536 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.539 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.549 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-21T05:33:16.552 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T05:33:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-21T05:33:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T05:33:16.554 INFO:teuthology.run_tasks:Running task install... 2024-09-21T05:33:16.562 DEBUG:teuthology.task.install:project ceph 2024-09-21T05:33:16.562 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T05:33:16.563 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T05:33:16.563 INFO:teuthology.task.install:Using flavor: default 2024-09-21T05:33:16.568 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T05:33:16.568 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-21T05:33:16.568 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T05:33:16.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:33:16.570 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T05:33:16.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:33:16.571 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T05:33:16.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:33:16.720 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T05:33:16.720 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T05:33:16.732 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T05:33:16.732 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T05:33:16.736 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T05:33:16.736 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T05:33:16.831 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T05:33:16.831 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:33:16.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T05:33:16.839 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T05:33:16.839 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:33:16.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T05:33:16.848 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T05:33:16.849 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:33:16.849 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T05:33:16.859 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T05:33:16.859 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T05:33:16.864 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T05:33:16.864 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T05:33:16.875 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T05:33:16.875 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T05:33:16.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T05:33:16.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T05:33:16.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T05:33:17.003 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T05:33:17.004 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T05:33:17.016 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T05:33:17.067 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-21T05:33:17.068 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-21T05:33:17.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-21T05:33:17.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-21T05:33:17.080 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-09-21T05:33:17.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-09-21T05:33:17.293 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-21T05:33:17.304 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-09-21T05:33:17.310 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-21T05:33:17.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T05:33:17.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T05:33:17.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T05:33:17.885 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 294 kB/s | 84 kB 00:00 2024-09-21T05:33:17.885 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-21T05:33:17.906 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-09-21T05:33:18.190 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 102 kB/s | 24 kB 00:00 2024-09-21T05:33:18.194 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 90 kB/s | 24 kB 00:00 2024-09-21T05:33:18.252 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 74 kB/s | 24 kB 00:00 2024-09-21T05:33:18.482 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 7.5 kB/s | 1.9 kB 00:00 2024-09-21T05:33:18.503 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-09-21T05:33:18.528 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-21T05:33:19.311 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T05:33:19.970 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.3 MB 00:01 2024-09-21T05:33:20.257 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T05:33:23.533 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-21T05:33:24.987 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-21T05:33:26.746 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 20 MB 00:05 2024-09-21T05:33:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-21T05:33:31.322 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-21T05:33:31.725 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.5 MB 00:04 2024-09-21T05:33:32.822 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-21T05:33:33.222 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-21T05:33:33.726 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 19 kB 00:00 2024-09-21T05:33:33.751 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T05:33:34.293 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-21T05:33:34.629 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T05:33:39.366 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T05:33:39.690 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-21T05:33:40.609 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-21T05:33:41.493 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.494 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.505 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T05:33:41.506 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.507 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T05:33:41.507 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.509 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T05:33:41.511 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.512 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.574 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T05:33:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T05:33:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T05:33:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T05:33:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T05:33:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T05:33:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T05:33:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T05:33:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T05:33:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T05:33:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T05:33:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T05:33:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout:Install 149 Packages 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 8 Packages 2024-09-21T05:33:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:33:41.613 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 228 M 2024-09-21T05:33:41.613 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T05:33:41.809 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.809 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.821 INFO:teuthology.orchestra.run.smithi181.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T05:33:41.822 INFO:teuthology.orchestra.run.smithi181.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T05:33:41.823 INFO:teuthology.orchestra.run.smithi181.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T05:33:41.823 INFO:teuthology.orchestra.run.smithi181.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.825 INFO:teuthology.orchestra.run.smithi181.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T05:33:41.827 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.828 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T05:33:41.890 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T05:33:41.917 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T05:33:41.917 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T05:33:41.917 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T05:33:41.917 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T05:33:41.918 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T05:33:41.919 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T05:33:41.920 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T05:33:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T05:33:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T05:33:41.923 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T05:33:41.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T05:33:41.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T05:33:41.926 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T05:33:41.927 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T05:33:41.928 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T05:33:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Install 149 Packages 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 8 Packages 2024-09-21T05:33:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:33:41.931 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 228 M 2024-09-21T05:33:41.931 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T05:33:42.275 INFO:teuthology.orchestra.run.smithi181.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 44 kB/s | 6.6 kB 00:00 2024-09-21T05:33:42.459 INFO:teuthology.orchestra.run.smithi129.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 48 kB/s | 6.6 kB 00:00 2024-09-21T05:33:42.467 INFO:teuthology.orchestra.run.smithi181.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 4.4 MB/s | 873 kB 00:00 2024-09-21T05:33:42.542 INFO:teuthology.orchestra.run.smithi181.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 1.9 MB/s | 148 kB 00:00 2024-09-21T05:33:42.643 INFO:teuthology.orchestra.run.smithi181.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-21T05:33:42.827 INFO:teuthology.orchestra.run.smithi181.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 8.1 MB/s | 2.3 MB 00:00 2024-09-21T05:33:42.868 INFO:teuthology.orchestra.run.smithi181.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 6.7 MB/s | 1.5 MB 00:00 2024-09-21T05:33:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 2.0 MB/s | 873 kB 00:00 2024-09-21T05:33:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:42.899 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T05:33:42.913 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T05:33:42.914 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T05:33:42.915 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T05:33:42.916 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T05:33:42.918 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T05:33:42.921 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T05:33:42.922 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T05:33:42.989 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T05:33:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T05:33:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T05:33:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T05:33:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T05:33:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T05:33:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T05:33:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T05:33:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T05:33:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T05:33:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T05:33:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T05:33:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:Install 149 Packages 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-09-21T05:33:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:33:43.021 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 228 M 2024-09-21T05:33:43.021 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-21T05:33:43.227 INFO:teuthology.orchestra.run.smithi129.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 434 kB/s | 148 kB 00:00 2024-09-21T05:33:43.327 INFO:teuthology.orchestra.run.smithi181.stdout:(7/157): ceph-common-19.3.0-5120.g18432218.el9. 17 MB/s | 20 MB 00:01 2024-09-21T05:33:43.402 INFO:teuthology.orchestra.run.smithi181.stdout:(8/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 8.6 MB/s | 5.0 MB 00:00 2024-09-21T05:33:43.461 INFO:teuthology.orchestra.run.smithi181.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 433 kB/s | 25 kB 00:00 2024-09-21T05:33:43.619 INFO:teuthology.orchestra.run.smithi129.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 4.3 MB/s | 5.5 MB 00:01 2024-09-21T05:33:43.704 INFO:teuthology.orchestra.run.smithi028.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-09-21T05:33:43.786 INFO:teuthology.orchestra.run.smithi129.stdout:(5/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-09-21T05:33:43.811 INFO:teuthology.orchestra.run.smithi181.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 22 MB/s | 10 MB 00:00 2024-09-21T05:33:43.986 INFO:teuthology.orchestra.run.smithi181.stdout:(11/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 16 MB/s | 18 MB 00:01 2024-09-21T05:33:44.011 INFO:teuthology.orchestra.run.smithi181.stdout:(12/157): cephfs-mirror-19.3.0-5120.g18432218.e 1.1 MB/s | 226 kB 00:00 2024-09-21T05:33:44.070 INFO:teuthology.orchestra.run.smithi181.stdout:(13/157): cephfs-shell-19.3.0-5120.g18432218.el 325 kB/s | 27 kB 00:00 2024-09-21T05:33:44.095 INFO:teuthology.orchestra.run.smithi181.stdout:(14/157): libcephfs-devel-19.3.0-5120.g18432218 379 kB/s | 32 kB 00:00 2024-09-21T05:33:44.154 INFO:teuthology.orchestra.run.smithi028.stdout:(2/157): ceph-base-19.3.0-5120.g18432218.el9.x8 7.1 MB/s | 5.5 MB 00:00 2024-09-21T05:33:44.195 INFO:teuthology.orchestra.run.smithi181.stdout:(15/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 766 kB 00:00 2024-09-21T05:33:44.221 INFO:teuthology.orchestra.run.smithi181.stdout:(16/157): libcephsqlite-19.3.0-5120.g18432218.e 1.2 MB/s | 158 kB 00:00 2024-09-21T05:33:44.221 INFO:teuthology.orchestra.run.smithi028.stdout:(3/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 1.6 MB/s | 873 kB 00:00 2024-09-21T05:33:44.227 INFO:teuthology.orchestra.run.smithi129.stdout:(6/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-21T05:33:44.237 INFO:teuthology.orchestra.run.smithi181.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 3.0 MB/s | 127 kB 00:00 2024-09-21T05:33:44.246 INFO:teuthology.orchestra.run.smithi028.stdout:(4/157): ceph-immutable-object-cache-19.3.0-512 1.6 MB/s | 148 kB 00:00 2024-09-21T05:33:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 6.6 MB/s | 507 kB 00:00 2024-09-21T05:33:44.413 INFO:teuthology.orchestra.run.smithi181.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 386 kB/s | 45 kB 00:00 2024-09-21T05:33:44.514 INFO:teuthology.orchestra.run.smithi181.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.5 MB/s | 148 kB 00:00 2024-09-21T05:33:44.605 INFO:teuthology.orchestra.run.smithi181.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 1.7 MB/s | 161 kB 00:00 2024-09-21T05:33:44.621 INFO:teuthology.orchestra.run.smithi028.stdout:(5/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 4.0 MB/s | 1.5 MB 00:00 2024-09-21T05:33:44.678 INFO:teuthology.orchestra.run.smithi129.stdout:(7/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 1.6 MB/s | 2.3 MB 00:01 2024-09-21T05:33:44.739 INFO:teuthology.orchestra.run.smithi181.stdout:(22/157): python3-rados-19.3.0-5120.g18432218.e 2.4 MB/s | 321 kB 00:00 2024-09-21T05:33:44.814 INFO:teuthology.orchestra.run.smithi181.stdout:(23/157): librgw2-19.3.0-5120.g18432218.el9.x86 9.2 MB/s | 5.3 MB 00:00 2024-09-21T05:33:44.840 INFO:teuthology.orchestra.run.smithi181.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 3.0 MB/s | 307 kB 00:00 2024-09-21T05:33:44.890 INFO:teuthology.orchestra.run.smithi181.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.3 MB/s | 99 kB 00:00 2024-09-21T05:33:44.915 INFO:teuthology.orchestra.run.smithi181.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.1 MB/s | 86 kB 00:00 2024-09-21T05:33:44.990 INFO:teuthology.orchestra.run.smithi181.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 2.2 MB/s | 170 kB 00:00 2024-09-21T05:33:45.057 INFO:teuthology.orchestra.run.smithi181.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 514 kB/s | 34 kB 00:00 2024-09-21T05:33:45.063 INFO:teuthology.orchestra.run.smithi028.stdout:(6/157): ceph-common-19.3.0-5120.g18432218.el9. 12 MB/s | 20 MB 00:01 2024-09-21T05:33:45.113 INFO:teuthology.orchestra.run.smithi028.stdout:(7/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-09-21T05:33:45.133 INFO:teuthology.orchestra.run.smithi181.stdout:(29/157): rbd-mirror-19.3.0-5120.g18432218.el9. 13 MB/s | 3.1 MB 00:00 2024-09-21T05:33:45.168 INFO:teuthology.orchestra.run.smithi181.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 1.4 MB/s | 154 kB 00:00 2024-09-21T05:33:45.705 INFO:teuthology.orchestra.run.smithi028.stdout:(8/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 4.6 MB/s | 5.0 MB 00:01 2024-09-21T05:33:45.745 INFO:teuthology.orchestra.run.smithi129.stdout:(8/157): ceph-osd-19.3.0-5120.g18432218.el9.x86 12 MB/s | 18 MB 00:01 2024-09-21T05:33:45.780 INFO:teuthology.orchestra.run.smithi028.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 336 kB/s | 25 kB 00:00 2024-09-21T05:33:45.828 INFO:teuthology.orchestra.run.smithi129.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 303 kB/s | 25 kB 00:00 2024-09-21T05:33:45.909 INFO:teuthology.orchestra.run.smithi181.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 9.9 MB/s | 7.4 MB 00:00 2024-09-21T05:33:46.026 INFO:teuthology.orchestra.run.smithi181.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 2.4 MB/s | 292 kB 00:00 2024-09-21T05:33:46.135 INFO:teuthology.orchestra.run.smithi181.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 455 kB/s | 49 kB 00:00 2024-09-21T05:33:46.227 INFO:teuthology.orchestra.run.smithi181.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 180 kB/s | 16 kB 00:00 2024-09-21T05:33:46.327 INFO:teuthology.orchestra.run.smithi181.stdout:(35/157): ceph-volume-19.3.0-5120.g18432218.el9 2.8 MB/s | 285 kB 00:00 2024-09-21T05:33:46.544 INFO:teuthology.orchestra.run.smithi181.stdout:(36/157): ceph-mgr-dashboard-19.3.0-5120.g18432 10 MB/s | 14 MB 00:01 2024-09-21T05:33:46.603 INFO:teuthology.orchestra.run.smithi181.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 317 kB/s | 18 kB 00:00 2024-09-21T05:33:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:(10/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 11 MB/s | 18 MB 00:01 2024-09-21T05:33:46.695 INFO:teuthology.orchestra.run.smithi181.stdout:(38/157): fuse-2.9.9-16.el9.x86_64.rpm 871 kB/s | 80 kB 00:00 2024-09-21T05:33:46.756 INFO:teuthology.orchestra.run.smithi028.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T05:33:46.787 INFO:teuthology.orchestra.run.smithi181.stdout:(39/157): libgfortran-11.5.0-2.el9.x86_64.rpm 8.5 MB/s | 801 kB 00:00 2024-09-21T05:33:46.829 INFO:teuthology.orchestra.run.smithi181.stdout:(40/157): libquadmath-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-21T05:33:46.862 INFO:teuthology.orchestra.run.smithi181.stdout:(41/157): cephadm-19.3.0-5120.g18432218.el9.noa 1.4 MB/s | 784 kB 00:00 2024-09-21T05:33:46.888 INFO:teuthology.orchestra.run.smithi181.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-09-21T05:33:46.889 INFO:teuthology.orchestra.run.smithi028.stdout:(12/157): ceph-radosgw-19.3.0-5120.g18432218.el 5.9 MB/s | 10 MB 00:01 2024-09-21T05:33:46.906 INFO:teuthology.orchestra.run.smithi028.stdout:(13/157): cephfs-shell-19.3.0-5120.g18432218.el 180 kB/s | 27 kB 00:00 2024-09-21T05:33:46.930 INFO:teuthology.orchestra.run.smithi181.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T05:33:46.940 INFO:teuthology.orchestra.run.smithi028.stdout:(14/157): libcephfs-devel-19.3.0-5120.g18432218 636 kB/s | 32 kB 00:00 2024-09-21T05:33:46.971 INFO:teuthology.orchestra.run.smithi181.stdout:(44/157): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-21T05:33:47.007 INFO:teuthology.orchestra.run.smithi028.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.3 MB/s | 158 kB 00:00 2024-09-21T05:33:47.013 INFO:teuthology.orchestra.run.smithi181.stdout:(45/157): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-09-21T05:33:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:(16/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 766 kB 00:00 2024-09-21T05:33:47.055 INFO:teuthology.orchestra.run.smithi181.stdout:(46/157): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-09-21T05:33:47.057 INFO:teuthology.orchestra.run.smithi028.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 2.5 MB/s | 127 kB 00:00 2024-09-21T05:33:47.081 INFO:teuthology.orchestra.run.smithi181.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-09-21T05:33:47.099 INFO:teuthology.orchestra.run.smithi028.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 7.4 MB/s | 507 kB 00:00 2024-09-21T05:33:47.166 INFO:teuthology.orchestra.run.smithi028.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 676 kB/s | 45 kB 00:00 2024-09-21T05:33:47.233 INFO:teuthology.orchestra.run.smithi028.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 2.2 MB/s | 148 kB 00:00 2024-09-21T05:33:47.300 INFO:teuthology.orchestra.run.smithi028.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 2.4 MB/s | 161 kB 00:00 2024-09-21T05:33:47.322 INFO:teuthology.orchestra.run.smithi181.stdout:(48/157): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-09-21T05:33:47.356 INFO:teuthology.orchestra.run.smithi181.stdout:(49/157): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 943 kB 00:00 2024-09-21T05:33:47.375 INFO:teuthology.orchestra.run.smithi028.stdout:(22/157): python3-rados-19.3.0-5120.g18432218.e 4.2 MB/s | 321 kB 00:00 2024-09-21T05:33:47.382 INFO:teuthology.orchestra.run.smithi181.stdout:(50/157): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-09-21T05:33:47.407 INFO:teuthology.orchestra.run.smithi181.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 433 kB/s | 22 kB 00:00 2024-09-21T05:33:47.440 INFO:teuthology.orchestra.run.smithi181.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-09-21T05:33:47.442 INFO:teuthology.orchestra.run.smithi028.stdout:(23/157): python3-rbd-19.3.0-5120.g18432218.el9 4.5 MB/s | 307 kB 00:00 2024-09-21T05:33:47.466 INFO:teuthology.orchestra.run.smithi181.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-09-21T05:33:47.509 INFO:teuthology.orchestra.run.smithi028.stdout:(24/157): python3-rgw-19.3.0-5120.g18432218.el9 1.5 MB/s | 99 kB 00:00 2024-09-21T05:33:47.516 INFO:teuthology.orchestra.run.smithi181.stdout:(54/157): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-09-21T05:33:47.575 INFO:teuthology.orchestra.run.smithi181.stdout:(55/157): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-09-21T05:33:47.576 INFO:teuthology.orchestra.run.smithi028.stdout:(25/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.3 MB/s | 86 kB 00:00 2024-09-21T05:33:47.633 INFO:teuthology.orchestra.run.smithi181.stdout:(56/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.9 MB/s | 171 kB 00:00 2024-09-21T05:33:47.692 INFO:teuthology.orchestra.run.smithi181.stdout:(57/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-21T05:33:47.693 INFO:teuthology.orchestra.run.smithi028.stdout:(26/157): librgw2-19.3.0-5120.g18432218.el9.x86 8.3 MB/s | 5.3 MB 00:00 2024-09-21T05:33:47.742 INFO:teuthology.orchestra.run.smithi181.stdout:(58/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-09-21T05:33:47.743 INFO:teuthology.orchestra.run.smithi028.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-21T05:33:47.794 INFO:teuthology.orchestra.run.smithi028.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 687 kB/s | 34 kB 00:00 2024-09-21T05:33:47.801 INFO:teuthology.orchestra.run.smithi181.stdout:(59/157): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-09-21T05:33:47.852 INFO:teuthology.orchestra.run.smithi028.stdout:(29/157): rbd-mirror-19.3.0-5120.g18432218.el9. 11 MB/s | 3.1 MB 00:00 2024-09-21T05:33:47.877 INFO:teuthology.orchestra.run.smithi028.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 1.8 MB/s | 154 kB 00:00 2024-09-21T05:33:48.012 INFO:teuthology.orchestra.run.smithi129.stdout:(10/157): ceph-test-19.3.0-5120.g18432218.el9.x 18 MB/s | 40 MB 00:02 2024-09-21T05:33:48.079 INFO:teuthology.orchestra.run.smithi129.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T05:33:48.110 INFO:teuthology.orchestra.run.smithi181.stdout:(60/157): libtool-2.4.6-46.el9.x86_64.rpm 1.8 MB/s | 581 kB 00:00 2024-09-21T05:33:48.137 INFO:teuthology.orchestra.run.smithi129.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 463 kB/s | 27 kB 00:00 2024-09-21T05:33:48.168 INFO:teuthology.orchestra.run.smithi181.stdout:(61/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 420 kB/s | 24 kB 00:00 2024-09-21T05:33:48.196 INFO:teuthology.orchestra.run.smithi129.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 545 kB/s | 32 kB 00:00 2024-09-21T05:33:48.218 INFO:teuthology.orchestra.run.smithi181.stdout:(62/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 954 kB/s | 47 kB 00:00 2024-09-21T05:33:48.312 INFO:teuthology.orchestra.run.smithi129.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.4 MB/s | 766 kB 00:00 2024-09-21T05:33:48.380 INFO:teuthology.orchestra.run.smithi129.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.3 MB/s | 158 kB 00:00 2024-09-21T05:33:48.385 INFO:teuthology.orchestra.run.smithi181.stdout:(63/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-21T05:33:48.438 INFO:teuthology.orchestra.run.smithi129.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 2.1 MB/s | 127 kB 00:00 2024-09-21T05:33:48.497 INFO:teuthology.orchestra.run.smithi129.stdout:(17/157): libradosstriper1-19.3.0-5120.g1843221 8.5 MB/s | 507 kB 00:00 2024-09-21T05:33:48.502 INFO:teuthology.orchestra.run.smithi181.stdout:(64/157): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-21T05:33:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 11 MB/s | 7.4 MB 00:00 2024-09-21T05:33:48.553 INFO:teuthology.orchestra.run.smithi181.stdout:(65/157): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-21T05:33:48.605 INFO:teuthology.orchestra.run.smithi028.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 3.3 MB/s | 292 kB 00:00 2024-09-21T05:33:48.681 INFO:teuthology.orchestra.run.smithi028.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 658 kB/s | 49 kB 00:00 2024-09-21T05:33:48.694 INFO:teuthology.orchestra.run.smithi181.stdout:(66/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.4 MB/s | 3.0 MB 00:01 2024-09-21T05:33:48.747 INFO:teuthology.orchestra.run.smithi028.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 247 kB/s | 16 kB 00:00 2024-09-21T05:33:48.806 INFO:teuthology.orchestra.run.smithi028.stdout:(35/157): ceph-volume-19.3.0-5120.g18432218.el9 4.8 MB/s | 285 kB 00:00 2024-09-21T05:33:48.814 INFO:teuthology.orchestra.run.smithi129.stdout:(18/157): librgw2-19.3.0-5120.g18432218.el9.x86 17 MB/s | 5.3 MB 00:00 2024-09-21T05:33:48.864 INFO:teuthology.orchestra.run.smithi129.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 908 kB/s | 45 kB 00:00 2024-09-21T05:33:48.956 INFO:teuthology.orchestra.run.smithi028.stdout:(36/157): ceph-mgr-dashboard-19.3.0-5120.g18432 13 MB/s | 14 MB 00:01 2024-09-21T05:33:48.998 INFO:teuthology.orchestra.run.smithi028.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 4.0 MB/s | 784 kB 00:00 2024-09-21T05:33:49.015 INFO:teuthology.orchestra.run.smithi028.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 315 kB/s | 18 kB 00:00 2024-09-21T05:33:49.081 INFO:teuthology.orchestra.run.smithi129.stdout:(20/157): ceph-common-19.3.0-5120.g18432218.el9 2.9 MB/s | 20 MB 00:06 2024-09-21T05:33:49.098 INFO:teuthology.orchestra.run.smithi129.stdout:(21/157): python3-ceph-common-19.3.0-5120.g1843 635 kB/s | 148 kB 00:00 2024-09-21T05:33:49.182 INFO:teuthology.orchestra.run.smithi028.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 435 kB/s | 80 kB 00:00 2024-09-21T05:33:49.207 INFO:teuthology.orchestra.run.smithi129.stdout:(22/157): ceph-radosgw-19.3.0-5120.g18432218.el 2.3 MB/s | 10 MB 00:04 2024-09-21T05:33:49.232 INFO:teuthology.orchestra.run.smithi129.stdout:(23/157): python3-cephfs-19.3.0-5120.g18432218. 1.0 MB/s | 161 kB 00:00 2024-09-21T05:33:49.257 INFO:teuthology.orchestra.run.smithi129.stdout:(24/157): python3-rados-19.3.0-5120.g18432218.e 2.0 MB/s | 321 kB 00:00 2024-09-21T05:33:49.265 INFO:teuthology.orchestra.run.smithi028.stdout:(40/157): libquadmath-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-21T05:33:49.274 INFO:teuthology.orchestra.run.smithi129.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 2.3 MB/s | 99 kB 00:00 2024-09-21T05:33:49.300 INFO:teuthology.orchestra.run.smithi129.stdout:(26/157): python3-rbd-19.3.0-5120.g18432218.el9 3.2 MB/s | 307 kB 00:00 2024-09-21T05:33:49.307 INFO:teuthology.orchestra.run.smithi028.stdout:(41/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-21T05:33:49.342 INFO:teuthology.orchestra.run.smithi129.stdout:(27/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.0 MB/s | 86 kB 00:00 2024-09-21T05:33:49.358 INFO:teuthology.orchestra.run.smithi028.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-21T05:33:49.383 INFO:teuthology.orchestra.run.smithi129.stdout:(28/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 2.0 MB/s | 170 kB 00:00 2024-09-21T05:33:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T05:33:49.409 INFO:teuthology.orchestra.run.smithi129.stdout:(29/157): ceph-grafana-dashboards-19.3.0-5120.g 513 kB/s | 34 kB 00:00 2024-09-21T05:33:49.442 INFO:teuthology.orchestra.run.smithi028.stdout:(44/157): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-09-21T05:33:49.451 INFO:teuthology.orchestra.run.smithi129.stdout:(30/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 2.3 MB/s | 154 kB 00:00 2024-09-21T05:33:49.467 INFO:teuthology.orchestra.run.smithi028.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 135 kB 00:00 2024-09-21T05:33:49.526 INFO:teuthology.orchestra.run.smithi028.stdout:(46/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-21T05:33:49.542 INFO:teuthology.orchestra.run.smithi028.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-09-21T05:33:49.768 INFO:teuthology.orchestra.run.smithi129.stdout:(31/157): rbd-mirror-19.3.0-5120.g18432218.el9. 6.3 MB/s | 3.1 MB 00:00 2024-09-21T05:33:49.853 INFO:teuthology.orchestra.run.smithi181.stdout:(67/157): ceph-test-19.3.0-5120.g18432218.el9.x 6.2 MB/s | 40 MB 00:06 2024-09-21T05:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:(48/157): ceph-test-19.3.0-5120.g18432218.el9.x 9.7 MB/s | 40 MB 00:04 2024-09-21T05:33:49.951 INFO:teuthology.orchestra.run.smithi129.stdout:(32/157): ceph-mgr-dashboard-19.3.0-5120.g18432 26 MB/s | 14 MB 00:00 2024-09-21T05:33:49.978 INFO:teuthology.orchestra.run.smithi129.stdout:(33/157): ceph-mgr-modules-core-19.3.0-5120.g18 1.4 MB/s | 292 kB 00:00 2024-09-21T05:33:49.993 INFO:teuthology.orchestra.run.smithi129.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 1.2 MB/s | 49 kB 00:00 2024-09-21T05:33:50.010 INFO:teuthology.orchestra.run.smithi129.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 492 kB/s | 16 kB 00:00 2024-09-21T05:33:50.018 INFO:teuthology.orchestra.run.smithi028.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 920 kB/s | 107 kB 00:00 2024-09-21T05:33:50.044 INFO:teuthology.orchestra.run.smithi129.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 5.5 MB/s | 285 kB 00:00 2024-09-21T05:33:50.051 INFO:teuthology.orchestra.run.smithi028.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 689 kB 00:00 2024-09-21T05:33:50.077 INFO:teuthology.orchestra.run.smithi028.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 370 kB/s | 22 kB 00:00 2024-09-21T05:33:50.078 INFO:teuthology.orchestra.run.smithi129.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 553 kB/s | 18 kB 00:00 2024-09-21T05:33:50.110 INFO:teuthology.orchestra.run.smithi028.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 1.6 MB/s | 943 kB 00:00 2024-09-21T05:33:50.136 INFO:teuthology.orchestra.run.smithi028.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-09-21T05:33:50.153 INFO:teuthology.orchestra.run.smithi181.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-09-21T05:33:50.161 INFO:teuthology.orchestra.run.smithi028.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-21T05:33:50.161 INFO:teuthology.orchestra.run.smithi129.stdout:(38/157): cephadm-19.3.0-5120.g18432218.el9.noa 5.1 MB/s | 784 kB 00:00 2024-09-21T05:33:50.186 INFO:teuthology.orchestra.run.smithi028.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-09-21T05:33:50.211 INFO:teuthology.orchestra.run.smithi028.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-09-21T05:33:50.236 INFO:teuthology.orchestra.run.smithi028.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-09-21T05:33:50.247 INFO:teuthology.orchestra.run.smithi129.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 473 kB/s | 80 kB 00:00 2024-09-21T05:33:50.312 INFO:teuthology.orchestra.run.smithi028.stdout:(58/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-09-21T05:33:50.337 INFO:teuthology.orchestra.run.smithi028.stdout:(59/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-21T05:33:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:(40/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2024-09-21T05:33:50.362 INFO:teuthology.orchestra.run.smithi028.stdout:(60/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-09-21T05:33:50.387 INFO:teuthology.orchestra.run.smithi129.stdout:(41/157): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-21T05:33:50.421 INFO:teuthology.orchestra.run.smithi028.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-21T05:33:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-09-21T05:33:50.447 INFO:teuthology.orchestra.run.smithi028.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 293 kB/s | 24 kB 00:00 2024-09-21T05:33:50.471 INFO:teuthology.orchestra.run.smithi028.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-21T05:33:50.488 INFO:teuthology.orchestra.run.smithi129.stdout:(43/157): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-09-21T05:33:50.505 INFO:teuthology.orchestra.run.smithi028.stdout:(64/157): libtool-2.4.6-46.el9.x86_64.rpm 4.0 MB/s | 581 kB 00:00 2024-09-21T05:33:50.518 INFO:teuthology.orchestra.run.smithi129.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-21T05:33:50.529 INFO:teuthology.orchestra.run.smithi181.stdout:(69/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-09-21T05:33:50.539 INFO:teuthology.orchestra.run.smithi028.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-21T05:33:50.564 INFO:teuthology.orchestra.run.smithi028.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-21T05:33:50.569 INFO:teuthology.orchestra.run.smithi129.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-09-21T05:33:50.589 INFO:teuthology.orchestra.run.smithi028.stdout:(67/157): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-21T05:33:50.602 INFO:teuthology.orchestra.run.smithi129.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-21T05:33:50.636 INFO:teuthology.orchestra.run.smithi129.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-09-21T05:33:50.637 INFO:teuthology.orchestra.run.smithi181.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 1.2 MB/s | 137 kB 00:00 2024-09-21T05:33:50.663 INFO:teuthology.orchestra.run.smithi181.stdout:(71/157): postgresql-contrib-13.16-1.el9.x86_64 1.7 MB/s | 910 kB 00:00 2024-09-21T05:33:50.673 INFO:teuthology.orchestra.run.smithi028.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T05:33:50.715 INFO:teuthology.orchestra.run.smithi028.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-21T05:33:50.729 INFO:teuthology.orchestra.run.smithi181.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.3 MB/s | 4.9 MB 00:02 2024-09-21T05:33:50.748 INFO:teuthology.orchestra.run.smithi028.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-09-21T05:33:50.773 INFO:teuthology.orchestra.run.smithi028.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-21T05:33:50.813 INFO:teuthology.orchestra.run.smithi181.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-09-21T05:33:50.840 INFO:teuthology.orchestra.run.smithi028.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 16 MB/s | 4.9 MB 00:00 2024-09-21T05:33:50.853 INFO:teuthology.orchestra.run.smithi129.stdout:(48/157): ceph-mgr-diskprediction-local-19.3.0- 5.3 MB/s | 7.4 MB 00:01 2024-09-21T05:33:50.882 INFO:teuthology.orchestra.run.smithi028.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-09-21T05:33:50.907 INFO:teuthology.orchestra.run.smithi028.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-21T05:33:50.920 INFO:teuthology.orchestra.run.smithi129.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-21T05:33:50.962 INFO:teuthology.orchestra.run.smithi129.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 523 kB/s | 22 kB 00:00 2024-09-21T05:33:50.987 INFO:teuthology.orchestra.run.smithi129.stdout:(51/157): boost-program-options-1.75.0-8.el9.x8 799 kB/s | 107 kB 00:00 2024-09-21T05:33:50.999 INFO:teuthology.orchestra.run.smithi028.stdout:(75/157): postgresql-server-13.16-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-09-21T05:33:51.012 INFO:teuthology.orchestra.run.smithi129.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-09-21T05:33:51.025 INFO:teuthology.orchestra.run.smithi028.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 524 kB/s | 61 kB 00:00 2024-09-21T05:33:51.029 INFO:teuthology.orchestra.run.smithi129.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 779 kB/s | 33 kB 00:00 2024-09-21T05:33:51.050 INFO:teuthology.orchestra.run.smithi028.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-21T05:33:51.055 INFO:teuthology.orchestra.run.smithi129.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-09-21T05:33:51.072 INFO:teuthology.orchestra.run.smithi129.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 434 kB/s | 18 kB 00:00 2024-09-21T05:33:51.075 INFO:teuthology.orchestra.run.smithi028.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-21T05:33:51.100 INFO:teuthology.orchestra.run.smithi028.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-09-21T05:33:51.125 INFO:teuthology.orchestra.run.smithi028.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-21T05:33:51.151 INFO:teuthology.orchestra.run.smithi028.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-21T05:33:51.155 INFO:teuthology.orchestra.run.smithi129.stdout:(56/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.0 MB/s | 171 kB 00:00 2024-09-21T05:33:51.172 INFO:teuthology.orchestra.run.smithi129.stdout:(57/157): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-09-21T05:33:51.176 INFO:teuthology.orchestra.run.smithi028.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-09-21T05:33:51.213 INFO:teuthology.orchestra.run.smithi181.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-21T05:33:51.260 INFO:teuthology.orchestra.run.smithi028.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-21T05:33:51.272 INFO:teuthology.orchestra.run.smithi181.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-09-21T05:33:51.285 INFO:teuthology.orchestra.run.smithi028.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-09-21T05:33:51.289 INFO:teuthology.orchestra.run.smithi129.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-21T05:33:51.310 INFO:teuthology.orchestra.run.smithi028.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-21T05:33:51.322 INFO:teuthology.orchestra.run.smithi129.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-09-21T05:33:51.335 INFO:teuthology.orchestra.run.smithi028.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T05:33:51.361 INFO:teuthology.orchestra.run.smithi028.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-09-21T05:33:51.386 INFO:teuthology.orchestra.run.smithi028.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-09-21T05:33:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-21T05:33:51.414 INFO:teuthology.orchestra.run.smithi181.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-21T05:33:51.437 INFO:teuthology.orchestra.run.smithi028.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-21T05:33:51.462 INFO:teuthology.orchestra.run.smithi028.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T05:33:51.487 INFO:teuthology.orchestra.run.smithi028.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-09-21T05:33:51.489 INFO:teuthology.orchestra.run.smithi129.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-09-21T05:33:51.515 INFO:teuthology.orchestra.run.smithi129.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 3.0 MB/s | 581 kB 00:00 2024-09-21T05:33:51.540 INFO:teuthology.orchestra.run.smithi129.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 486 kB/s | 24 kB 00:00 2024-09-21T05:33:51.562 INFO:teuthology.orchestra.run.smithi028.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-21T05:33:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-09-21T05:33:51.565 INFO:teuthology.orchestra.run.smithi129.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-09-21T05:33:51.587 INFO:teuthology.orchestra.run.smithi028.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-09-21T05:33:51.613 INFO:teuthology.orchestra.run.smithi028.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-09-21T05:33:51.614 INFO:teuthology.orchestra.run.smithi181.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 515 kB/s | 26 kB 00:00 2024-09-21T05:33:51.638 INFO:teuthology.orchestra.run.smithi028.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-21T05:33:51.658 INFO:teuthology.orchestra.run.smithi129.stdout:(64/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-09-21T05:33:51.663 INFO:teuthology.orchestra.run.smithi028.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-09-21T05:33:51.664 INFO:teuthology.orchestra.run.smithi181.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-21T05:33:51.683 INFO:teuthology.orchestra.run.smithi129.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-21T05:33:51.689 INFO:teuthology.orchestra.run.smithi028.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-21T05:33:51.708 INFO:teuthology.orchestra.run.smithi129.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-21T05:33:51.714 INFO:teuthology.orchestra.run.smithi028.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-21T05:33:51.739 INFO:teuthology.orchestra.run.smithi028.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T05:33:51.764 INFO:teuthology.orchestra.run.smithi028.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-21T05:33:51.773 INFO:teuthology.orchestra.run.smithi181.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-21T05:33:51.789 INFO:teuthology.orchestra.run.smithi028.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T05:33:51.792 INFO:teuthology.orchestra.run.smithi129.stdout:(67/157): openblas-0.3.26-2.el9.x86_64.rpm 287 kB/s | 38 kB 00:00 2024-09-21T05:33:51.815 INFO:teuthology.orchestra.run.smithi028.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-09-21T05:33:51.823 INFO:teuthology.orchestra.run.smithi181.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-09-21T05:33:51.907 INFO:teuthology.orchestra.run.smithi028.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-21T05:33:52.074 INFO:teuthology.orchestra.run.smithi028.stdout:(105/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-21T05:33:52.116 INFO:teuthology.orchestra.run.smithi028.stdout:(106/157): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-09-21T05:33:52.141 INFO:teuthology.orchestra.run.smithi028.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 847 kB/s | 19 kB 00:00 2024-09-21T05:33:52.174 INFO:teuthology.orchestra.run.smithi028.stdout:(108/157): gperftools-libs-2.9.1-3.el9.x86_64.r 3.1 MB/s | 308 kB 00:00 2024-09-21T05:33:52.208 INFO:teuthology.orchestra.run.smithi028.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-09-21T05:33:52.233 INFO:teuthology.orchestra.run.smithi028.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-09-21T05:33:52.267 INFO:teuthology.orchestra.run.smithi028.stdout:(111/157): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T05:33:52.300 INFO:teuthology.orchestra.run.smithi028.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 4.7 MB/s | 151 kB 00:00 2024-09-21T05:33:52.325 INFO:teuthology.orchestra.run.smithi028.stdout:(113/157): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-09-21T05:33:52.375 INFO:teuthology.orchestra.run.smithi129.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-09-21T05:33:52.384 INFO:teuthology.orchestra.run.smithi028.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-21T05:33:52.401 INFO:teuthology.orchestra.run.smithi028.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-21T05:33:52.426 INFO:teuthology.orchestra.run.smithi028.stdout:(116/157): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-09-21T05:33:52.443 INFO:teuthology.orchestra.run.smithi028.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 770 kB/s | 32 kB 00:00 2024-09-21T05:33:52.460 INFO:teuthology.orchestra.run.smithi028.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 422 kB/s | 14 kB 00:00 2024-09-21T05:33:52.486 INFO:teuthology.orchestra.run.smithi028.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-21T05:33:52.519 INFO:teuthology.orchestra.run.smithi028.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-21T05:33:52.544 INFO:teuthology.orchestra.run.smithi028.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-21T05:33:52.569 INFO:teuthology.orchestra.run.smithi028.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-21T05:33:52.603 INFO:teuthology.orchestra.run.smithi028.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-21T05:33:52.629 INFO:teuthology.orchestra.run.smithi028.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-09-21T05:33:52.695 INFO:teuthology.orchestra.run.smithi028.stdout:(125/157): libarrow-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-09-21T05:33:52.720 INFO:teuthology.orchestra.run.smithi028.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-09-21T05:33:52.746 INFO:teuthology.orchestra.run.smithi028.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 935 kB/s | 23 kB 00:00 2024-09-21T05:33:52.771 INFO:teuthology.orchestra.run.smithi028.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-09-21T05:33:52.796 INFO:teuthology.orchestra.run.smithi028.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 794 kB/s | 20 kB 00:00 2024-09-21T05:33:52.821 INFO:teuthology.orchestra.run.smithi028.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T05:33:52.847 INFO:teuthology.orchestra.run.smithi028.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T05:33:52.859 INFO:teuthology.orchestra.run.smithi129.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 1.8 MB/s | 910 kB 00:00 2024-09-21T05:33:52.880 INFO:teuthology.orchestra.run.smithi028.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-21T05:33:52.914 INFO:teuthology.orchestra.run.smithi028.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-21T05:33:52.939 INFO:teuthology.orchestra.run.smithi028.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-09-21T05:33:52.943 INFO:teuthology.orchestra.run.smithi129.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 1.6 MB/s | 137 kB 00:00 2024-09-21T05:33:52.990 INFO:teuthology.orchestra.run.smithi028.stdout:(135/157): python3-grpcio-1.46.7-10.el9.x86_64. 4.6 MB/s | 2.0 MB 00:00 2024-09-21T05:33:53.081 INFO:teuthology.orchestra.run.smithi028.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 634 kB/s | 90 kB 00:00 2024-09-21T05:33:53.085 INFO:teuthology.orchestra.run.smithi129.stdout:(71/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-09-21T05:33:53.115 INFO:teuthology.orchestra.run.smithi028.stdout:(137/157): python3-kubernetes-26.1.0-2.el9.noar 3.0 MB/s | 1.0 MB 00:00 2024-09-21T05:33:53.140 INFO:teuthology.orchestra.run.smithi028.stdout:(138/157): python3-pyperclip-1.8.2-2.el9.noarch 484 kB/s | 28 kB 00:00 2024-09-21T05:33:53.165 INFO:teuthology.orchestra.run.smithi028.stdout:(139/157): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-09-21T05:33:53.191 INFO:teuthology.orchestra.run.smithi028.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T05:33:53.216 INFO:teuthology.orchestra.run.smithi028.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 717 kB/s | 36 kB 00:00 2024-09-21T05:33:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 3.2 MB/s | 4.9 MB 00:01 2024-09-21T05:33:53.241 INFO:teuthology.orchestra.run.smithi028.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-21T05:33:53.266 INFO:teuthology.orchestra.run.smithi028.stdout:(143/157): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-09-21T05:33:53.277 INFO:teuthology.orchestra.run.smithi129.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-21T05:33:53.292 INFO:teuthology.orchestra.run.smithi028.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-09-21T05:33:53.317 INFO:teuthology.orchestra.run.smithi028.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-21T05:33:53.342 INFO:teuthology.orchestra.run.smithi028.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-21T05:33:53.368 INFO:teuthology.orchestra.run.smithi028.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T05:33:53.393 INFO:teuthology.orchestra.run.smithi028.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-21T05:33:53.443 INFO:teuthology.orchestra.run.smithi028.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T05:33:53.469 INFO:teuthology.orchestra.run.smithi129.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-21T05:33:53.519 INFO:teuthology.orchestra.run.smithi129.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-21T05:33:53.574 INFO:teuthology.orchestra.run.smithi181.stdout:(82/157): postgresql-server-13.16-1.el9.x86_64. 2.0 MB/s | 5.8 MB 00:02 2024-09-21T05:33:53.636 INFO:teuthology.orchestra.run.smithi129.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-21T05:33:53.663 INFO:teuthology.orchestra.run.smithi028.stdout:(150/157): librbd1-19.3.0-5120.g18432218.el9.x8 12 MB/s | 3.2 MB 00:00 2024-09-21T05:33:53.718 INFO:teuthology.orchestra.run.smithi028.stdout:(151/157): librados2-19.3.0-5120.g18432218.el9. 9.7 MB/s | 3.4 MB 00:00 2024-09-21T05:33:53.724 INFO:teuthology.orchestra.run.smithi181.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-09-21T05:33:53.753 INFO:teuthology.orchestra.run.smithi129.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-09-21T05:33:53.799 INFO:teuthology.orchestra.run.smithi181.stdout:(84/157): python3-babel-2.9.1-2.el9.noarch.rpm 2.0 MB/s | 6.0 MB 00:02 2024-09-21T05:33:53.803 INFO:teuthology.orchestra.run.smithi129.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-09-21T05:33:53.853 INFO:teuthology.orchestra.run.smithi129.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-21T05:33:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 289 kB/s | 41 kB 00:00 2024-09-21T05:33:53.883 INFO:teuthology.orchestra.run.smithi181.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 926 kB/s | 77 kB 00:00 2024-09-21T05:33:53.908 INFO:teuthology.orchestra.run.smithi181.stdout:(86/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-09-21T05:33:53.933 INFO:teuthology.orchestra.run.smithi181.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-09-21T05:33:53.937 INFO:teuthology.orchestra.run.smithi129.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-21T05:33:53.979 INFO:teuthology.orchestra.run.smithi129.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 832 kB/s | 35 kB 00:00 2024-09-21T05:33:53.994 INFO:teuthology.orchestra.run.smithi028.stdout:(153/157): NetworkManager-libnm-1.51.0-1.el9.x8 5.5 MB/s | 1.8 MB 00:00 2024-09-21T05:33:54.019 INFO:teuthology.orchestra.run.smithi028.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.5 MB/s | 249 kB 00:00 2024-09-21T05:33:54.042 INFO:teuthology.orchestra.run.smithi181.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-21T05:33:54.109 INFO:teuthology.orchestra.run.smithi181.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 1.3 MB/s | 269 kB 00:00 2024-09-21T05:33:54.159 INFO:teuthology.orchestra.run.smithi181.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T05:33:54.186 INFO:teuthology.orchestra.run.smithi028.stdout:(155/157): NetworkManager-1.51.0-1.el9.x86_64.r 3.1 MB/s | 2.3 MB 00:00 2024-09-21T05:33:54.209 INFO:teuthology.orchestra.run.smithi181.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-21T05:33:54.268 INFO:teuthology.orchestra.run.smithi181.stdout:(92/157): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 279 kB 00:00 2024-09-21T05:33:54.327 INFO:teuthology.orchestra.run.smithi181.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 717 kB/s | 42 kB 00:00 2024-09-21T05:33:54.427 INFO:teuthology.orchestra.run.smithi181.stdout:(94/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.0 MB/s | 5.1 MB 00:02 2024-09-21T05:33:54.452 INFO:teuthology.orchestra.run.smithi181.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 353 kB/s | 44 kB 00:00 2024-09-21T05:33:54.502 INFO:teuthology.orchestra.run.smithi181.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-09-21T05:33:54.504 INFO:teuthology.orchestra.run.smithi129.stdout:(82/157): postgresql-server-13.16-1.el9.x86_64. 3.7 MB/s | 5.8 MB 00:01 2024-09-21T05:33:54.553 INFO:teuthology.orchestra.run.smithi181.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-21T05:33:54.578 INFO:teuthology.orchestra.run.smithi181.stdout:(98/157): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-21T05:33:54.645 INFO:teuthology.orchestra.run.smithi181.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 868 kB/s | 58 kB 00:00 2024-09-21T05:33:54.671 INFO:teuthology.orchestra.run.smithi129.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-09-21T05:33:54.697 INFO:teuthology.orchestra.run.smithi129.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-09-21T05:33:54.711 INFO:teuthology.orchestra.run.smithi028.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 1.9 MB/s | 1.4 MB 00:00 2024-09-21T05:33:54.729 INFO:teuthology.orchestra.run.smithi181.stdout:(100/157): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-09-21T05:33:54.747 INFO:teuthology.orchestra.run.smithi129.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T05:33:54.754 INFO:teuthology.orchestra.run.smithi181.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 882 kB/s | 96 kB 00:00 2024-09-21T05:33:54.772 INFO:teuthology.orchestra.run.smithi129.stdout:(86/157): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-09-21T05:33:54.789 INFO:teuthology.orchestra.run.smithi129.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-21T05:33:54.794 INFO:teuthology.orchestra.run.smithi028.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.1 MB/s | 2.4 MB 00:00 2024-09-21T05:33:54.801 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-21T05:33:54.801 INFO:teuthology.orchestra.run.smithi028.stdout:Total 19 MB/s | 228 MB 00:11 2024-09-21T05:33:54.813 INFO:teuthology.orchestra.run.smithi181.stdout:(102/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 767 kB/s | 45 kB 00:00 2024-09-21T05:33:54.838 INFO:teuthology.orchestra.run.smithi181.stdout:(103/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 585 kB/s | 64 kB 00:00 2024-09-21T05:33:54.856 INFO:teuthology.orchestra.run.smithi129.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-09-21T05:33:54.881 INFO:teuthology.orchestra.run.smithi129.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-09-21T05:33:54.923 INFO:teuthology.orchestra.run.smithi129.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-21T05:33:54.949 INFO:teuthology.orchestra.run.smithi129.stdout:(91/157): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-09-21T05:33:54.965 INFO:teuthology.orchestra.run.smithi129.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-09-21T05:33:55.007 INFO:teuthology.orchestra.run.smithi129.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-21T05:33:55.049 INFO:teuthology.orchestra.run.smithi129.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-21T05:33:55.142 INFO:teuthology.orchestra.run.smithi129.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-21T05:33:55.184 INFO:teuthology.orchestra.run.smithi129.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-09-21T05:33:55.226 INFO:teuthology.orchestra.run.smithi129.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-21T05:33:55.342 INFO:teuthology.orchestra.run.smithi129.stdout:(98/157): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-21T05:33:55.430 INFO:teuthology.orchestra.run.smithi181.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-09-21T05:33:55.480 INFO:teuthology.orchestra.run.smithi181.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T05:33:55.493 INFO:teuthology.orchestra.run.smithi129.stdout:(99/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-09-21T05:33:55.514 INFO:teuthology.orchestra.run.smithi181.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.0 MB/s | 308 kB 00:00 2024-09-21T05:33:55.518 INFO:teuthology.orchestra.run.smithi129.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 329 kB/s | 58 kB 00:00 2024-09-21T05:33:55.535 INFO:teuthology.orchestra.run.smithi129.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 2.2 MB/s | 96 kB 00:00 2024-09-21T05:33:55.539 INFO:teuthology.orchestra.run.smithi181.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-09-21T05:33:55.569 INFO:teuthology.orchestra.run.smithi129.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T05:33:55.594 INFO:teuthology.orchestra.run.smithi129.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 763 kB/s | 45 kB 00:00 2024-09-21T05:33:55.647 INFO:teuthology.orchestra.run.smithi181.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T05:33:55.673 INFO:teuthology.orchestra.run.smithi181.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-21T05:33:55.698 INFO:teuthology.orchestra.run.smithi181.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T05:33:55.723 INFO:teuthology.orchestra.run.smithi181.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T05:33:55.749 INFO:teuthology.orchestra.run.smithi181.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-21T05:33:55.782 INFO:teuthology.orchestra.run.smithi181.stdout:(113/157): protobuf-compiler-3.14.0-13.el9.x86_ 914 kB/s | 863 kB 00:00 2024-09-21T05:33:55.833 INFO:teuthology.orchestra.run.smithi181.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-21T05:33:55.866 INFO:teuthology.orchestra.run.smithi181.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-21T05:33:55.878 INFO:teuthology.orchestra.run.smithi129.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-09-21T05:33:55.892 INFO:teuthology.orchestra.run.smithi181.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-21T05:33:55.916 INFO:teuthology.orchestra.run.smithi181.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 571 kB/s | 14 kB 00:00 2024-09-21T05:33:55.928 INFO:teuthology.orchestra.run.smithi129.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T05:33:55.950 INFO:teuthology.orchestra.run.smithi181.stdout:(118/157): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-09-21T05:33:55.953 INFO:teuthology.orchestra.run.smithi129.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 12 MB/s | 308 kB 00:00 2024-09-21T05:33:55.976 INFO:teuthology.orchestra.run.smithi181.stdout:(119/157): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-09-21T05:33:55.979 INFO:teuthology.orchestra.run.smithi129.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 772 kB/s | 19 kB 00:00 2024-09-21T05:33:56.009 INFO:teuthology.orchestra.run.smithi181.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-21T05:33:56.035 INFO:teuthology.orchestra.run.smithi181.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-21T05:33:56.042 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-21T05:33:56.060 INFO:teuthology.orchestra.run.smithi181.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-21T05:33:56.086 INFO:teuthology.orchestra.run.smithi181.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-09-21T05:33:56.102 INFO:teuthology.orchestra.run.smithi129.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 36 MB/s | 4.4 MB 00:00 2024-09-21T05:33:56.110 INFO:teuthology.orchestra.run.smithi181.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-21T05:33:56.129 INFO:teuthology.orchestra.run.smithi129.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 992 kB/s | 26 kB 00:00 2024-09-21T05:33:56.136 INFO:teuthology.orchestra.run.smithi181.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 708 kB/s | 18 kB 00:00 2024-09-21T05:33:56.154 INFO:teuthology.orchestra.run.smithi129.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T05:33:56.161 INFO:teuthology.orchestra.run.smithi181.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-09-21T05:33:56.180 INFO:teuthology.orchestra.run.smithi129.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T05:33:56.186 INFO:teuthology.orchestra.run.smithi181.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 782 kB/s | 19 kB 00:00 2024-09-21T05:33:56.205 INFO:teuthology.orchestra.run.smithi129.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-21T05:33:56.212 INFO:teuthology.orchestra.run.smithi181.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 786 kB/s | 20 kB 00:00 2024-09-21T05:33:56.212 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-21T05:33:56.212 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-21T05:33:56.255 INFO:teuthology.orchestra.run.smithi129.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-09-21T05:33:56.270 INFO:teuthology.orchestra.run.smithi181.stdout:(129/157): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-09-21T05:33:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-21T05:33:56.295 INFO:teuthology.orchestra.run.smithi181.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T05:33:56.306 INFO:teuthology.orchestra.run.smithi129.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-21T05:33:56.321 INFO:teuthology.orchestra.run.smithi181.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T05:33:56.323 INFO:teuthology.orchestra.run.smithi129.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.9 MB/s | 32 kB 00:00 2024-09-21T05:33:56.340 INFO:teuthology.orchestra.run.smithi129.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 843 kB/s | 14 kB 00:00 2024-09-21T05:33:56.346 INFO:teuthology.orchestra.run.smithi181.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-21T05:33:56.373 INFO:teuthology.orchestra.run.smithi129.stdout:(118/157): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-09-21T05:33:56.380 INFO:teuthology.orchestra.run.smithi181.stdout:(133/157): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-09-21T05:33:56.405 INFO:teuthology.orchestra.run.smithi181.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-09-21T05:33:56.407 INFO:teuthology.orchestra.run.smithi129.stdout:(119/157): python3-cherrypy-18.6.1-2.el9.noarch 10 MB/s | 358 kB 00:00 2024-09-21T05:33:56.430 INFO:teuthology.orchestra.run.smithi181.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-21T05:33:56.441 INFO:teuthology.orchestra.run.smithi129.stdout:(120/157): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-09-21T05:33:56.456 INFO:teuthology.orchestra.run.smithi181.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-21T05:33:56.474 INFO:teuthology.orchestra.run.smithi129.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-09-21T05:33:56.481 INFO:teuthology.orchestra.run.smithi181.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-21T05:33:56.517 INFO:teuthology.orchestra.run.smithi129.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 2.8 MB/s | 220 kB 00:00 2024-09-21T05:33:56.526 INFO:teuthology.orchestra.run.smithi181.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 437 kB/s | 31 kB 00:00 2024-09-21T05:33:56.548 INFO:teuthology.orchestra.run.smithi181.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.7 MB/s | 59 kB 00:00 2024-09-21T05:33:56.551 INFO:teuthology.orchestra.run.smithi129.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-21T05:33:56.573 INFO:teuthology.orchestra.run.smithi181.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-09-21T05:33:56.576 INFO:teuthology.orchestra.run.smithi129.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-21T05:33:56.599 INFO:teuthology.orchestra.run.smithi181.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-09-21T05:33:56.601 INFO:teuthology.orchestra.run.smithi129.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 703 kB/s | 18 kB 00:00 2024-09-21T05:33:56.624 INFO:teuthology.orchestra.run.smithi181.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-21T05:33:56.626 INFO:teuthology.orchestra.run.smithi129.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-09-21T05:33:56.649 INFO:teuthology.orchestra.run.smithi181.stdout:(143/157): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-21T05:33:56.666 INFO:teuthology.orchestra.run.smithi181.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-09-21T05:33:56.668 INFO:teuthology.orchestra.run.smithi129.stdout:(127/157): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-09-21T05:33:56.683 INFO:teuthology.orchestra.run.smithi181.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-09-21T05:33:56.693 INFO:teuthology.orchestra.run.smithi129.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-09-21T05:33:56.708 INFO:teuthology.orchestra.run.smithi181.stdout:(146/157): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-21T05:33:56.719 INFO:teuthology.orchestra.run.smithi129.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-21T05:33:56.734 INFO:teuthology.orchestra.run.smithi181.stdout:(147/157): python3-werkzeug-2.0.3-3.el9.1.noarc 4.9 MB/s | 427 kB 00:00 2024-09-21T05:33:56.744 INFO:teuthology.orchestra.run.smithi129.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T05:33:56.770 INFO:teuthology.orchestra.run.smithi129.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T05:33:56.793 INFO:teuthology.orchestra.run.smithi181.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-21T05:33:56.795 INFO:teuthology.orchestra.run.smithi129.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-21T05:33:56.828 INFO:teuthology.orchestra.run.smithi129.stdout:(133/157): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-09-21T05:33:56.854 INFO:teuthology.orchestra.run.smithi129.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 661 kB/s | 16 kB 00:00 2024-09-21T05:33:56.879 INFO:teuthology.orchestra.run.smithi129.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-21T05:33:56.904 INFO:teuthology.orchestra.run.smithi129.stdout:(136/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-21T05:33:56.930 INFO:teuthology.orchestra.run.smithi129.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 559 kB/s | 28 kB 00:00 2024-09-21T05:33:56.955 INFO:teuthology.orchestra.run.smithi129.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-21T05:33:56.980 INFO:teuthology.orchestra.run.smithi129.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T05:33:57.006 INFO:teuthology.orchestra.run.smithi129.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-21T05:33:57.031 INFO:teuthology.orchestra.run.smithi129.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 708 kB/s | 36 kB 00:00 2024-09-21T05:33:57.043 INFO:teuthology.orchestra.run.smithi181.stdout:(149/157): librbd1-19.3.0-5120.g18432218.el9.x8 13 MB/s | 3.2 MB 00:00 2024-09-21T05:33:57.056 INFO:teuthology.orchestra.run.smithi129.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-21T05:33:57.082 INFO:teuthology.orchestra.run.smithi129.stdout:(143/157): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-21T05:33:57.107 INFO:teuthology.orchestra.run.smithi129.stdout:(144/157): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T05:33:57.195 INFO:teuthology.orchestra.run.smithi181.stdout:(150/157): librados2-19.3.0-5120.g18432218.el9. 8.3 MB/s | 3.4 MB 00:00 2024-09-21T05:33:57.196 INFO:teuthology.orchestra.run.smithi129.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 598 kB/s | 20 kB 00:00 2024-09-21T05:33:57.196 INFO:teuthology.orchestra.run.smithi129.stdout:(146/157): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-09-21T05:33:57.196 INFO:teuthology.orchestra.run.smithi129.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T05:33:57.233 INFO:teuthology.orchestra.run.smithi129.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-21T05:33:57.252 INFO:teuthology.orchestra.run.smithi181.stdout:(151/157): NetworkManager-1.51.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-09-21T05:33:57.294 INFO:teuthology.orchestra.run.smithi181.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 983 kB/s | 41 kB 00:00 2024-09-21T05:33:57.336 INFO:teuthology.orchestra.run.smithi181.stdout:(153/157): NetworkManager-libnm-1.51.0-1.el9.x8 9.4 MB/s | 1.8 MB 00:00 2024-09-21T05:33:57.361 INFO:teuthology.orchestra.run.smithi181.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 3.6 MB/s | 249 kB 00:00 2024-09-21T05:33:57.403 INFO:teuthology.orchestra.run.smithi181.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 20 MB/s | 1.4 MB 00:00 2024-09-21T05:33:57.453 INFO:teuthology.orchestra.run.smithi181.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-09-21T05:33:57.475 INFO:teuthology.orchestra.run.smithi129.stdout:(149/157): librbd1-19.3.0-5120.g18432218.el9.x8 13 MB/s | 3.2 MB 00:00 2024-09-21T05:33:57.676 INFO:teuthology.orchestra.run.smithi129.stdout:(150/157): NetworkManager-1.51.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-09-21T05:33:57.768 INFO:teuthology.orchestra.run.smithi129.stdout:(151/157): librados2-19.3.0-5120.g18432218.el9. 5.9 MB/s | 3.4 MB 00:00 2024-09-21T05:33:57.826 INFO:teuthology.orchestra.run.smithi129.stdout:(152/157): NetworkManager-libnm-1.51.0-1.el9.x8 12 MB/s | 1.8 MB 00:00 2024-09-21T05:33:57.851 INFO:teuthology.orchestra.run.smithi129.stdout:(153/157): NetworkManager-team-1.51.0-1.el9.x86 490 kB/s | 41 kB 00:00 2024-09-21T05:33:57.918 INFO:teuthology.orchestra.run.smithi129.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 2.6 MB/s | 249 kB 00:00 2024-09-21T05:33:57.961 INFO:teuthology.orchestra.run.smithi129.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 12 MB/s | 1.4 MB 00:00 2024-09-21T05:33:58.044 INFO:teuthology.orchestra.run.smithi129.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-09-21T05:33:58.478 INFO:teuthology.orchestra.run.smithi129.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.6 MB/s | 16 MB 00:03 2024-09-21T05:33:58.482 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T05:33:58.482 INFO:teuthology.orchestra.run.smithi129.stdout:Total 14 MB/s | 228 MB 00:16 2024-09-21T05:33:58.676 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-21T05:33:58.677 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-21T05:33:59.707 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T05:33:59.858 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T05:33:59.859 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T05:34:01.672 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T05:34:01.673 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T05:34:02.786 INFO:teuthology.orchestra.run.smithi181.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.9 MB/s | 16 MB 00:08 2024-09-21T05:34:02.791 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T05:34:02.792 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 228 MB 00:20 2024-09-21T05:34:04.028 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T05:34:04.180 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T05:34:04.180 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T05:34:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-21T05:34:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T05:34:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T05:34:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T05:34:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T05:34:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T05:34:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T05:34:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T05:34:06.000 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T05:34:06.001 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T05:34:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:34:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T05:34:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T05:34:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T05:34:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:07.275 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:07.395 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T05:34:07.503 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:07.563 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:07.801 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:07.913 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T05:34:07.951 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T05:34:07.986 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T05:34:08.070 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T05:34:08.120 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T05:34:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T05:34:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T05:34:08.206 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T05:34:08.245 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T05:34:08.272 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T05:34:08.290 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T05:34:08.328 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T05:34:08.344 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T05:34:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T05:34:08.395 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T05:34:08.416 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T05:34:08.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T05:34:08.441 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T05:34:08.469 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T05:34:08.483 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T05:34:08.553 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T05:34:08.587 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T05:34:08.631 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T05:34:08.668 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T05:34:08.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T05:34:08.791 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T05:34:08.818 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T05:34:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T05:34:08.934 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T05:34:09.069 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T05:34:09.069 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:09.094 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:09.184 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:09.215 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:34:09.244 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T05:34:09.399 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T05:34:09.455 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T05:34:09.483 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:09.588 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:09.617 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:09.627 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T05:34:09.680 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:09.702 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:09.716 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T05:34:09.722 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:09.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T05:34:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T05:34:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T05:34:09.870 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T05:34:10.004 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:10.053 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T05:34:10.109 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T05:34:10.142 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T05:34:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T05:34:10.246 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:10.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T05:34:10.327 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:10.340 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T05:34:10.378 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T05:34:10.414 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T05:34:10.461 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T05:34:10.496 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T05:34:10.540 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T05:34:10.562 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:10.582 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T05:34:10.667 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T05:34:10.707 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T05:34:10.724 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T05:34:10.741 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T05:34:10.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T05:34:10.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T05:34:10.832 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T05:34:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T05:34:10.908 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T05:34:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T05:34:10.951 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T05:34:10.984 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T05:34:11.016 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T05:34:11.084 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T05:34:11.137 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T05:34:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T05:34:11.168 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T05:34:11.203 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T05:34:11.207 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T05:34:11.240 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T05:34:11.264 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T05:34:11.284 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T05:34:11.315 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T05:34:11.359 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T05:34:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T05:34:11.387 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T05:34:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T05:34:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T05:34:11.514 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T05:34:11.555 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T05:34:11.575 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T05:34:11.592 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T05:34:11.625 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T05:34:11.665 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T05:34:11.704 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T05:34:11.709 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T05:34:11.737 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T05:34:11.763 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T05:34:11.780 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T05:34:11.813 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T05:34:11.855 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T05:34:11.889 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T05:34:11.980 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T05:34:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T05:34:12.067 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T05:34:12.175 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T05:34:12.227 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T05:34:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T05:34:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T05:34:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T05:34:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T05:34:12.448 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T05:34:12.485 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T05:34:12.527 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T05:34:12.573 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T05:34:12.611 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T05:34:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T05:34:12.698 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T05:34:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T05:34:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T05:34:12.755 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T05:34:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T05:34:12.845 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T05:34:12.919 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T05:34:12.992 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T05:34:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T05:34:13.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T05:34:13.040 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T05:34:13.064 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T05:34:13.073 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T05:34:13.115 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T05:34:13.147 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T05:34:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T05:34:13.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T05:34:13.190 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T05:34:13.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T05:34:13.232 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T05:34:13.233 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:13.291 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T05:34:13.359 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:13.367 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T05:34:13.381 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T05:34:13.385 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:13.410 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T05:34:13.417 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T05:34:13.451 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T05:34:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:34:13.535 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T05:34:13.553 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T05:34:13.585 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T05:34:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T05:34:13.616 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T05:34:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T05:34:13.651 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T05:34:13.675 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T05:34:13.688 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T05:34:13.711 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T05:34:13.745 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T05:34:13.757 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T05:34:13.772 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:13.871 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:13.894 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T05:34:13.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:13.925 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T05:34:13.953 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T05:34:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T05:34:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:13.987 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T05:34:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T05:34:14.026 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T05:34:14.058 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T05:34:14.088 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T05:34:14.122 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T05:34:14.156 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T05:34:14.193 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:14.197 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T05:34:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T05:34:14.319 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T05:34:14.345 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T05:34:14.397 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T05:34:14.440 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T05:34:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T05:34:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:14.608 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:14.722 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T05:34:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T05:34:14.883 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T05:34:14.941 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T05:34:14.974 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T05:34:15.011 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T05:34:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T05:34:15.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T05:34:15.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T05:34:15.169 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T05:34:15.208 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T05:34:15.252 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T05:34:15.285 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T05:34:15.312 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T05:34:15.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T05:34:15.451 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T05:34:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T05:34:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T05:34:15.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T05:34:15.636 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T05:34:15.680 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T05:34:15.709 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T05:34:15.772 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T05:34:15.789 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T05:34:15.823 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T05:34:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T05:34:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T05:34:15.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T05:34:15.896 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T05:34:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T05:34:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T05:34:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T05:34:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T05:34:16.022 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T05:34:16.027 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T05:34:16.067 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T05:34:16.084 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T05:34:16.148 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T05:34:16.187 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T05:34:16.349 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T05:34:16.371 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T05:34:16.402 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T05:34:16.405 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:16.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T05:34:16.506 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:16.527 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T05:34:16.563 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T05:34:16.604 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T05:34:16.697 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T05:34:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T05:34:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T05:34:16.806 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T05:34:16.848 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T05:34:17.016 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T05:34:17.086 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T05:34:17.090 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T05:34:17.116 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T05:34:17.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:17.177 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T05:34:17.188 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:17.218 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T05:34:17.250 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T05:34:17.250 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T05:34:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T05:34:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T05:34:17.389 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T05:34:17.437 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T05:34:17.473 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T05:34:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T05:34:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T05:34:17.695 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T05:34:17.735 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T05:34:17.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T05:34:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T05:34:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T05:34:18.042 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T05:34:18.101 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T05:34:18.125 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T05:34:18.201 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T05:34:18.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T05:34:18.374 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T05:34:18.402 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T05:34:18.436 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T05:34:18.475 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T05:34:18.480 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T05:34:18.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T05:34:18.537 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T05:34:18.572 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T05:34:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T05:34:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T05:34:18.604 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T05:34:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T05:34:18.646 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T05:34:18.680 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T05:34:18.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T05:34:18.821 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T05:34:18.852 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T05:34:18.955 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T05:34:18.964 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T05:34:19.008 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:19.012 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T05:34:19.100 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:19.130 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T05:34:19.165 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T05:34:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:19.673 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T05:34:19.703 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:19.725 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T05:34:19.774 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:19.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T05:34:19.805 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T05:34:19.809 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T05:34:19.917 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T05:34:19.973 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T05:34:20.027 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T05:34:20.113 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T05:34:20.145 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T05:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:20.319 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T05:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:20.351 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T05:34:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:20.426 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T05:34:20.426 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T05:34:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:20.427 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T05:34:20.469 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:21.065 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T05:34:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T05:34:22.038 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:22.068 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:22.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T05:34:22.471 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T05:34:22.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T05:34:22.532 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T05:34:22.638 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:22.680 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:22.773 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:22.773 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T05:34:22.773 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T05:34:22.773 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:22.807 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:22.856 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T05:34:22.904 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:22.994 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T05:34:23.017 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T05:34:23.053 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T05:34:23.547 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T05:34:23.573 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:23.636 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T05:34:23.658 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T05:34:25.949 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:25.988 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:26.552 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T05:34:26.591 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:26.687 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T05:34:26.687 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T05:34:26.688 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T05:34:26.688 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:26.718 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-21T05:34:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-21T05:34:32.741 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-21T05:34:32.741 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-21T05:34:32.741 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:32.928 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:32.928 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:32.928 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T05:34:32.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:34.610 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:34.610 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T05:34:34.610 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:34.610 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-09-21T05:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:34.872 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:34.874 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:34.875 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T05:34:34.875 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:34.875 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:34.889 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:34.889 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:34.890 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T05:34:34.890 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:35.506 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T05:34:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T05:34:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T05:34:35.833 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T05:34:35.876 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T05:34:35.958 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T05:34:36.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T05:34:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T05:34:36.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T05:34:36.291 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T05:34:36.318 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T05:34:36.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:36.511 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:36.734 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:36.755 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:36.757 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:36.758 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:36.758 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T05:34:36.758 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:36.758 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:36.758 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T05:34:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T05:34:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:37.227 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:37.250 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:37.250 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:37.250 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T05:34:37.251 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:37.251 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:37.251 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:37.357 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T05:34:37.414 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T05:34:37.469 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T05:34:37.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T05:34:37.703 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T05:34:37.781 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T05:34:37.877 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T05:34:38.026 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T05:34:38.061 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T05:34:38.093 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T05:34:38.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T05:34:38.506 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:38.537 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-21T05:34:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:38.569 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T05:34:38.635 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T05:34:38.660 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:38.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:38.738 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T05:34:38.738 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:38.738 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T05:34:38.738 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:40.314 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:40.333 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:40.359 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T05:34:40.359 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:40.359 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T05:34:40.359 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:40.359 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T05:34:40.360 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:40.370 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:40.583 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T05:34:40.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:41.076 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:41.099 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T05:34:41.099 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:41.099 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T05:34:41.099 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:41.100 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T05:34:41.100 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:41.206 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T05:34:41.254 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T05:34:41.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T05:34:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:41.495 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T05:34:41.542 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:41.542 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T05:34:41.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T05:34:41.700 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T05:34:41.711 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T05:34:41.739 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:41.764 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:41.764 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T05:34:41.764 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:41.764 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:41.839 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T05:34:41.867 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:41.884 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T05:34:41.916 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T05:34:41.917 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T05:34:41.944 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T05:34:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T05:34:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T05:34:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T05:34:42.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T05:34:42.524 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:42.878 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:42.909 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T05:34:43.022 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:43.067 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:43.067 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T05:34:43.067 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:43.067 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:43.133 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:43.133 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T05:34:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T05:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:43.210 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T05:34:43.236 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:43.259 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:43.260 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:43.260 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T05:34:43.260 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:43.260 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:43.260 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:43.303 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:43.315 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:43.414 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T05:34:43.472 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T05:34:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:43.568 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:43.568 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T05:34:43.568 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:43.568 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:44.439 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T05:34:44.465 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:44.488 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:44.489 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:44.489 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T05:34:44.489 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:44.489 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:44.489 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:44.532 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:44.555 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:44.555 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:44.555 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T05:34:44.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:44.599 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:44.622 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:44.623 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:44.623 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T05:34:44.623 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:44.623 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:44.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:44.942 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:44.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:34:45.686 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T05:34:45.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T05:34:46.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:46.881 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T05:34:47.056 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T05:34:47.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T05:34:47.115 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:47.116 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T05:34:47.116 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:47.116 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T05:34:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T05:34:47.278 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T05:34:47.323 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T05:34:47.815 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T05:34:47.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T05:34:47.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T05:34:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T05:34:47.964 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T05:34:48.200 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T05:34:48.244 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T05:34:48.274 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T05:34:48.286 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T05:34:48.304 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T05:34:48.328 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T05:34:48.359 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T05:34:48.359 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:48.360 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T05:34:48.360 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:48.360 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T05:34:48.360 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:48.395 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:48.418 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T05:34:48.418 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:48.418 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T05:34:48.418 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:48.463 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T05:34:48.486 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:48.830 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:48.853 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T05:34:48.853 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T05:34:48.853 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T05:34:48.853 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:48.853 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T05:34:48.854 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:34:48.922 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T05:34:48.963 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T05:34:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:49.003 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T05:34:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:49.035 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T05:34:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:49.063 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T05:34:49.243 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T05:34:49.285 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T05:34:49.315 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T05:34:49.345 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T05:34:49.399 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T05:34:49.911 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:49.946 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T05:34:49.956 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:49.976 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:50.017 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T05:34:50.070 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T05:34:50.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T05:34:50.195 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T05:34:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T05:34:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T05:34:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:50.330 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:50.331 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:50.414 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T05:34:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:50.700 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:50.768 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:50.795 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T05:34:50.878 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T05:34:50.919 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T05:34:51.001 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T05:34:51.041 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T05:34:51.065 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T05:34:51.092 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T05:34:51.092 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:51.176 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:51.176 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:51.260 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:51.285 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T05:34:51.290 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:51.290 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:51.470 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:51.471 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:51.571 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:52.826 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T05:34:52.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T05:34:52.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T05:34:52.956 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T05:34:52.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T05:34:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T05:34:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T05:34:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T05:34:53.273 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T05:34:53.320 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T05:34:53.940 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:53.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:54.005 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T05:34:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T05:34:54.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T05:34:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T05:34:55.085 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T05:34:55.142 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T05:34:55.167 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T05:34:55.193 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T05:34:55.193 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:55.277 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T05:34:55.277 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:55.361 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T05:34:55.386 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T05:34:55.392 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:55.392 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:55.572 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T05:34:55.572 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:55.656 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T05:34:55.720 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T05:34:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T05:34:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T05:34:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T05:34:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T05:34:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T05:34:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T05:34:56.961 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T05:34:56.962 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T05:34:56.963 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T05:34:56.964 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T05:34:56.965 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T05:34:56.966 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T05:34:56.967 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T05:34:56.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T05:34:56.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T05:34:56.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T05:34:56.971 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T05:34:56.972 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T05:34:56.973 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T05:34:56.973 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T05:34:56.973 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T05:34:57.686 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:57.686 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:34:57.687 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T05:34:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T05:34:57.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T05:34:57.701 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T05:34:57.702 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T05:34:57.703 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T05:34:57.704 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T05:34:57.705 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T05:34:57.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T05:34:57.707 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T05:34:57.708 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T05:34:57.709 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T05:34:57.710 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T05:34:57.710 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T05:34:57.710 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T05:34:59.616 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.617 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.618 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T05:34:59.619 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.620 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T05:34:59.621 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.622 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T05:34:59.623 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T05:34:59.624 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T05:34:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T05:34:59.626 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T05:34:59.627 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T05:34:59.628 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T05:34:59.628 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:34:59.628 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-21T05:35:00.013 DEBUG:teuthology.parallel:result is None 2024-09-21T05:35:00.420 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T05:35:00.421 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.422 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.423 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.424 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T05:35:00.425 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T05:35:00.426 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T05:35:00.427 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T05:35:00.428 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T05:35:00.429 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T05:35:00.430 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T05:35:00.431 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T05:35:00.432 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T05:35:00.433 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T05:35:00.433 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T05:35:00.433 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:35:00.433 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T05:35:00.806 DEBUG:teuthology.parallel:result is None 2024-09-21T05:35:01.932 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:35:01.932 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T05:35:01.932 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T05:35:01.932 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T05:35:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T05:35:01.934 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T05:35:01.935 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T05:35:01.935 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T05:35:01.946 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T05:35:01.946 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T05:35:01.946 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T05:35:01.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T05:35:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T05:35:01.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T05:35:01.950 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T05:35:01.951 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T05:35:01.952 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T05:35:01.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T05:35:01.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T05:35:01.955 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T05:35:01.956 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T05:35:01.957 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T05:35:01.957 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T05:35:01.957 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T05:35:01.957 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T05:35:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T05:35:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T05:35:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T05:35:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T05:35:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T05:35:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T05:35:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T05:35:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T05:35:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T05:35:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:35:04.594 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T05:35:04.978 DEBUG:teuthology.parallel:result is None 2024-09-21T05:35:04.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:05.131 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T05:35:05.157 INFO:teuthology.orchestra.run.smithi028.stdout:19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.158 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.158 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T05:35:05.160 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:05.303 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T05:35:05.329 INFO:teuthology.orchestra.run.smithi129.stdout:19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.329 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.329 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T05:35:05.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:05.477 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T05:35:05.501 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.501 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T05:35:05.501 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T05:35:05.503 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T05:35:05.503 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:35:05.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T05:35:05.537 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:35:05.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T05:35:05.570 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:35:05.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T05:35:05.598 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T05:35:05.599 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:35:05.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T05:35:05.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T05:35:05.698 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:35:05.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T05:35:05.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T05:35:05.796 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:35:05.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T05:35:05.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T05:35:05.889 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T05:35:05.889 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:35:05.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T05:35:05.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T05:35:05.991 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:35:05.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T05:35:06.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T05:35:06.089 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:35:06.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T05:35:06.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T05:35:06.181 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T05:35:06.182 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:35:06.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T05:35:06.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T05:35:06.282 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:35:06.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T05:35:06.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T05:35:06.380 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:35:06.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T05:35:06.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T05:35:06.474 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T05:35:06.585 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug', 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-21T05:35:06.585 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:06.585 INFO:tasks.cephadm:Cluster fsid is 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:35:06.585 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T05:35:06.586 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.129', 'mon.c': '172.21.15.181'} 2024-09-21T05:35:06.586 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-09-21T05:35:06.586 INFO:tasks.cephadm:First mgr is x 2024-09-21T05:35:06.586 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T05:35:06.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-09-21T05:35:06.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-09-21T05:35:06.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-09-21T05:35:06.673 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T05:35:06.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:06.819 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 00:04:17.484514', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 22:54:16.499090', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T05:35:06.966 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240920.212106-debug, sha1 1843221895adccfb3b692060251dd6a0183bc410 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:35:06.967 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T05:35:06.967 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T05:35:06.968 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:07.493 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 05:35 /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:07.493 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.003 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 05:35 /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.003 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.518 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 05:35 /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.518 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.538 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.558 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T05:35:08.601 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 on all hosts... 2024-09-21T05:35:08.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T05:35:08.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T05:35:08.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T05:35:10.213 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T05:35:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T05:35:10.263 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-21T05:35:44.627 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-21T05:35:47.660 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T05:35:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T05:35:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T05:35:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-09-21T05:35:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T05:35:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T05:35:47.661 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:36:03.251 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:36:03.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-09-21T05:36:03.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-09-21T05:36:03.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-09-21T05:36:03.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-09-21T05:36:03.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-09-21T05:36:03.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-09-21T05:36:03.473 INFO:tasks.cephadm:Writing seed config... 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T05:36:03.474 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-21T05:36:03.475 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-21T05:36:03.476 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-09-21T05:36:03.476 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-21T05:36:03.476 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T05:36:03.476 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-21T05:36:03.476 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:36:03.476 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T05:36:03.492 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 42500114-77db-11ef-baf5-efdc52797490 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T05:36:03.492 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service 2024-09-21T05:36:03.534 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x.service 2024-09-21T05:36:03.576 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T05:36:03.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 -v bootstrap --fsid 42500114-77db-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:36:03.788 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-21T05:36:03.788 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', '-v', 'bootstrap', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-09-21T05:36:03.788 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T05:36:03.788 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-09-21T05:36:03.813 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-21T05:36:03.813 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-09-21T05:36:03.813 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-09-21T05:36:03.821 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T05:36:03.821 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T05:36:03.828 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T05:36:03.828 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:36:03.835 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-21T05:36:03.841 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-21T05:36:03.842 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-21T05:36:03.842 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-09-21T05:36:03.864 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-21T05:36:03.864 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T05:36:03.864 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-09-21T05:36:03.864 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-09-21T05:36:03.871 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T05:36:03.871 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T05:36:03.877 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T05:36:03.877 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:36:03.884 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139771624184176 on /run/cephadm/42500114-77db-11ef-baf5-efdc52797490.lock 2024-09-21T05:36:03.891 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139771624184176 acquired on /run/cephadm/42500114-77db-11ef-baf5-efdc52797490.lock 2024-09-21T05:36:03.892 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-09-21T05:36:03.892 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-09-21T05:36:03.892 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-21T05:36:03.895 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-09-21T05:36:03.895 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-09-21T05:36:03.898 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T05:36:03.898 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T05:36:03.898 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-09-21T05:36:03.900 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T05:36:03.901 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-09-21T05:36:03.902 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-09-21T05:36:03.902 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T05:36:03.902 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T05:36:03.902 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T05:36:04.473 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T05:36:04.473 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T05:36:04.473 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T05:36:04.473 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:8425bc7408d91b180e7fdc62f23e7ae6603757f4f36e9969a4eaf9fd9416b764 2024-09-21T05:36:04.474 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T05:36:04.474 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T05:36:04.474 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T05:36:05.335 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T05:36:05.335 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T05:36:05.335 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T05:36:06.249 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-09-21T05:36:06.249 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-09-21T05:36:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBGW+5mKLFUIxAALDbTYgu0AaujSznpNyK2VQ== 2024-09-21T05:36:08.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBHW+5mIlq0HBAAosGJ8dqOlT2v0pvxhjQ95g== 2024-09-21T05:36:08.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBIW+5mJ4KoFhAAgeHPtnMOGapIBBPTe9YAJQ== 2024-09-21T05:36:08.939 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:36:09.769 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-09-21T05:36:10.717 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-09-21T05:36:10.925 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T05:36:11.072 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T05:36:11.255 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-42500114-77db-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490.target. 2024-09-21T05:36:11.255 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-42500114-77db-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490.target. 2024-09-21T05:36:11.503 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a 2024-09-21T05:36:11.503 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-21T05:36:11.677 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490.target.wants/ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490@.service. 2024-09-21T05:36:12.225 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T05:36:12.225 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-21T05:36:12.232 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T05:36:12.232 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:36:12.232 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-21T05:36:12.232 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T05:36:12.233 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-09-21T05:36:12.233 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-09-21T05:36:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:12 smithi028 systemd[1]: Started Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.546296s) 2024-09-21T05:36:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-09-21T05:36:13.196 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: monmap epoch 1 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: last_changed 2024-09-21T05:36:09.260492+0000 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: min_mon_release 19 (squid) 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: election_strategy: 1 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:36:13.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: fsmap 2024-09-21T05:36:13.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T05:36:13.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: mgrmap e1: no daemons active 2024-09-21T05:36:13.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:13 smithi028 ceph-mon[30859]: from='client.? 172.21.15.28:0/1091117860' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-21T05:36:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T05:36:14.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T05:36:14.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T05:36:14.238 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-09-21T05:36:14.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:14 smithi028 ceph-mon[30859]: from='client.? 172.21.15.28:0/3651227020' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T05:36:14.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:14 smithi028 ceph-mon[30859]: from='client.? 172.21.15.28:0/3651227020' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T05:36:15.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:15 smithi028 ceph-mon[30859]: from='client.? 172.21.15.28:0/63426380' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:36:19.460 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-09-21T05:36:19.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:19 smithi028 systemd[1]: Stopping Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:36:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a[30855]: 2024-09-21T05:36:20.753+0000 7fa8c5dbf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:36:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a[30855]: 2024-09-21T05:36:20.753+0000 7fa8c5dbf640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T05:36:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:20 smithi028 podman[31103]: 2024-09-21 05:36:20.824776037 +0000 UTC m=+1.094809506 container died 93de20a4bb78689ec623b92e9ed366a9432505e35c4e68b3fa0fa0dc37ab5b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:36:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:24 smithi028 podman[31103]: 2024-09-21 05:36:24.604933792 +0000 UTC m=+4.874967258 container cleanup 93de20a4bb78689ec623b92e9ed366a9432505e35c4e68b3fa0fa0dc37ab5b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T05:36:25.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:24 smithi028 bash[31103]: ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a 2024-09-21T05:36:26.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 podman[31118]: 2024-09-21 05:36:26.394712484 +0000 UTC m=+5.563302453 container remove 93de20a4bb78689ec623b92e9ed366a9432505e35c4e68b3fa0fa0dc37ab5b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T05:36:26.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T05:36:26.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 systemd[1]: Stopped Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:36:26.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 systemd[1]: Starting Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:36:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 podman[31209]: 2024-09-21 05:36:26.736917785 +0000 UTC m=+0.110293106 container create a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 podman[31209]: 2024-09-21 05:36:26.643155276 +0000 UTC m=+0.016530610 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 podman[31209]: 2024-09-21 05:36:26.897320785 +0000 UTC m=+0.270696107 container init a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 podman[31209]: 2024-09-21 05:36:26.900637997 +0000 UTC m=+0.274013328 container start a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: pidfile_write: ignore empty --pid-file 2024-09-21T05:36:26.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: load: jerasure load: lrc 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: RocksDB version: 7.9.2 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Git sha 0 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: DB SUMMARY 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: DB Session ID: H957BZV22HBP4JBO3OW6 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: CURRENT file: CURRENT 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T05:36:27.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95040 ; 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.error_if_exists: 0 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.create_if_missing: 0 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.paranoid_checks: 1 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.env: 0x56107ed31d60 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.info_log: 0x561085373360 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T05:36:27.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.statistics: (nil) 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.use_fsync: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_log_file_size: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_fallocate: 1 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.use_direct_reads: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T05:36:27.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.db_log_dir: 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.wal_dir: 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.write_buffer_manager: 0x561085377d60 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T05:36:27.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.unordered_write: 0 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.row_cache: None 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.wal_filter: None 2024-09-21T05:36:27.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.two_write_queues: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.wal_compression: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.atomic_flush: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.log_readahead_size: 0 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T05:36:27.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_background_jobs: 2 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_background_compactions: -1 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_subcompactions: 1 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T05:36:27.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_open_files: -1 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_background_flushes: -1 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Compression algorithms supported: 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kZSTD supported: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kXpressCompression supported: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kBZip2Compression supported: 0 2024-09-21T05:36:27.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kLZ4Compression supported: 1 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kZlibCompression supported: 1 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: kSnappyCompression supported: 1 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.merge_operator: 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_filter: None 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T05:36:27.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561085372fa0) 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-09-21T05:36:27.010 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x561081337610 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-09-21T05:36:27.011 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression: NoCompression 2024-09-21T05:36:27.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.num_levels: 7 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T05:36:27.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T05:36:27.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T05:36:27.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T05:36:27.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.inplace_update_support: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.bloom_locality: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.max_successive_merges: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T05:36:27.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.ttl: 2592000 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enable_blob_files: false 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.min_blob_size: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T05:36:27.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3113abd6-4d3f-4ab8-a000-afb53cdd5247 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726896986936896, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726896986938204, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91219, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 285, "table_properties": {"data_size": 89267, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11684, "raw_average_key_size": 47, "raw_value_size": 82663, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 244, "num_filter_entries": 244, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726896986, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3113abd6-4d3f-4ab8-a000-afb53cdd5247", "db_session_id": "H957BZV22HBP4JBO3OW6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726896986938309, "job": 1, "event": "recovery_finished"} 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561085396a80 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: DB pointer 0x56108613c000 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-09-21T05:36:27.019 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 90.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 90.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T05:36:27.020 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 103.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T05:36:27.021 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 23.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 23.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x561081337610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???) e1 preinit fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).mds e1 new map 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).mds e1 print_map 2024-09-21T05:36:27.022 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: btime 2024-09-21T05:36:12:178942+0000 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 bash[31209]: a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 systemd[1]: Started Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:36:27.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: monmap epoch 1 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: last_changed 2024-09-21T05:36:09.260492+0000 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: min_mon_release 19 (squid) 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: election_strategy: 1 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: fsmap 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T05:36:27.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:26 smithi028 ceph-mon[31227]: mgrmap e1: no daemons active 2024-09-21T05:36:27.983 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T05:36:28.004 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:36:28.004 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-09-21T05:36:28.005 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T05:36:28.005 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T05:36:28.266 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x 2024-09-21T05:36:28.266 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x.service: Unit ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x.service not loaded. 2024-09-21T05:36:28.419 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490.target.wants/ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x.service → /etc/systemd/system/ceph-42500114-77db-11ef-baf5-efdc52797490@.service. 2024-09-21T05:36:28.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:28 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1709736293' entity='client.admin' 2024-09-21T05:36:28.765 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:28 smithi028 podman[31441]: 2024-09-21 05:36:28.763830135 +0000 UTC m=+0.207113911 container create 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T05:36:29.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:28 smithi028 podman[31441]: 2024-09-21 05:36:28.991507477 +0000 UTC m=+0.434791264 container init 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T05:36:29.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:28 smithi028 podman[31441]: 2024-09-21 05:36:28.994456394 +0000 UTC m=+0.437740175 container start 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:36:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T05:36:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-21T05:36:29.099 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T05:36:29.099 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:36:29.099 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-21T05:36:29.099 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T05:36:29.108 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T05:36:29.108 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-21T05:36:29.116 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T05:36:29.116 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:36:29.116 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-21T05:36:29.116 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T05:36:29.116 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-09-21T05:36:29.117 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-09-21T05:36:29.336 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:29 smithi028 bash[31441]: 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b 2024-09-21T05:36:29.336 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:29 smithi028 systemd[1]: Started Ceph mgr.x for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:36:29.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1785612107' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T05:36:30.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-21T05:36:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T05:36:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T05:36:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T05:36:12:178942+0000", 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T05:36:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T05:36:12.179752+0000", 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:36:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-09-21T05:36:32.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:32 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/423433780' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T05:36:33.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T05:36:33.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T05:36:33.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T05:36:33.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T05:36:12:178942+0000", 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T05:36:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T05:36:12.179752+0000", 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T05:36:33.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:36:33.330 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-09-21T05:36:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: Activating manager daemon x 2024-09-21T05:36:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: mgrmap e2: x(active, starting, since 0.00356495s) 2024-09-21T05:36:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: Manager daemon x is now available 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' 2024-09-21T05:36:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:34 smithi028 ceph-mon[31227]: from='mgr.14100 172.21.15.28:0/1882388610' entity='mgr.x' 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T05:36:36.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:36.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T05:36:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T05:36:12:178942+0000", 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T05:36:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T05:36:12.179752+0000", 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:36:36.517 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-09-21T05:36:36.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:36 smithi028 ceph-mon[31227]: mgrmap e3: x(active, since 1.008s) 2024-09-21T05:36:36.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3361514492' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T05:36:37.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:37 smithi028 ceph-mon[31227]: mgrmap e4: x(active, since 2s) 2024-09-21T05:36:37.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:37 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2167200544' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T05:36:37.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T05:36:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T05:36:37.802 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T05:36:37.802 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-09-21T05:36:38.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1049120152' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T05:36:38.884 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setuser ceph since I am not root 2024-09-21T05:36:38.884 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setgroup ceph since I am not root 2024-09-21T05:36:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:39 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1049120152' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T05:36:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:39 smithi028 ceph-mon[31227]: mgrmap e5: x(active, since 4s) 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-21T05:36:40.459 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-09-21T05:36:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:40 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/703139038' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T05:36:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: Active manager daemon x restarted 2024-09-21T05:36:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: Activating manager daemon x 2024-09-21T05:36:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T05:36:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: mgrmap e6: x(active, starting, since 0.00396933s) 2024-09-21T05:36:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:36:44.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:36:44.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:36:44.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:36:44.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:36:44.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:43 smithi028 ceph-mon[31227]: Manager daemon x is now available 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-09-21T05:36:45.375 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying orchestrator module is enabled... 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: mgrmap e7: x(active, since 1.00679s) 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: Found migration_current of "None". Setting to last migration. 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:46.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:36:46.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:36:46.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:36:46.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:45 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:36:46.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:36:46.109+0000 7f6a294d6640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:05:36:46] ENGINE Error in HTTPServer.serve 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Traceback (most recent call last): 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._connections.run(self.expiration_interval) 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._run(expiration_interval) 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s = self.context.wrap_socket( 2024-09-21T05:36:46.562 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: return self.sslsocket_class._create( 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self.do_handshake() 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._sslobj.do_handshake() 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:36:46.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:36:47.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: mgrmap e8: x(active, since 2s) 2024-09-21T05:36:47.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: [21/Sep/2024:05:36:45] ENGINE Bus STARTING 2024-09-21T05:36:47.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1796349313' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T05:36:47.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: [21/Sep/2024:05:36:46] ENGINE Serving on https://172.21.15.28:7150 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: [21/Sep/2024:05:36:46] ENGINE Error in HTTPServer.serve 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:36:47.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: [21/Sep/2024:05:36:46] ENGINE Serving on http://172.21.15.28:8765 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: [21/Sep/2024:05:36:46] ENGINE Bus STARTED 2024-09-21T05:36:47.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:46 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:36:47.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T05:36:47.689 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-09-21T05:36:48.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:47 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1796349313' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T05:36:48.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:47 smithi028 ceph-mon[31227]: mgrmap e9: x(active, since 3s) 2024-09-21T05:36:49.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:49 smithi028 ceph-mon[31227]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:36:49.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:49 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:49.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:49 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:36:50.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T05:36:50.051 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-09-21T05:36:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:50 smithi028 ceph-mon[31227]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:36:51.514 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Generating public/private rsa key pair. 2024-09-21T05:36:51.514 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Your identification has been saved in /tmp/tmpfjnvkgk8/key 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Your public key has been saved in /tmp/tmpfjnvkgk8/key.pub 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: The key fingerprint is: 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: SHA256:DNTbBYnZKT1EkwFjhTUABwCiFBqo2UrDgfYG9xNb9J0 ceph-42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: The key's randomart image is: 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: +---[RSA 3072]----+ 2024-09-21T05:36:51.515 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |=oo...==*&X* | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |** . o +B.Ooo | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |*o= . = = E | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |o+.o + o. . | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |..o . S | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: |. | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: | | 2024-09-21T05:36:51.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: | | 2024-09-21T05:36:51.517 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: | | 2024-09-21T05:36:51.517 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:36:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: +----[SHA256]-----+ 2024-09-21T05:36:52.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:52 smithi028 ceph-mon[31227]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:36:52.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:52 smithi028 ceph-mon[31227]: Generating ssh key... 2024-09-21T05:36:52.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:52 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:52.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:52 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:52.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:36:52.696 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T05:36:52.697 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T05:36:52.704 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-09-21T05:36:53.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:53 smithi028 ceph-mon[31227]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:36:54.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:54 smithi028 ceph-mon[31227]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:36:55.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:55 smithi028 ceph-mon[31227]: Deploying cephadm binary to smithi028 2024-09-21T05:36:57.945 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-09-21T05:36:57.945 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-09-21T05:36:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:58 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:36:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:58 smithi028 ceph-mon[31227]: Added host smithi028 2024-09-21T05:36:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:58 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:00.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:59 smithi028 ceph-mon[31227]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:00.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:59 smithi028 ceph-mon[31227]: Saving service mon spec with placement count:5 2024-09-21T05:37:00.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:59 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:00.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:36:59 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:00.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T05:37:00.100 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-09-21T05:37:01.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T05:37:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:01 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:01 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:02 smithi028 ceph-mon[31227]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:02 smithi028 ceph-mon[31227]: Saving service mgr spec with placement count:2 2024-09-21T05:37:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:02 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2447031660' entity='client.admin' 2024-09-21T05:37:04.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:03 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:04.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:03 smithi028 ceph-mon[31227]: from='mgr.14118 172.21.15.28:0/1635933660' entity='mgr.x' 2024-09-21T05:37:04.333 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-09-21T05:37:04.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:04 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2119251304' entity='client.admin' 2024-09-21T05:37:04.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:04 smithi028 ceph-mon[31227]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:05 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1116031598' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T05:37:06.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setuser ceph since I am not root 2024-09-21T05:37:06.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setgroup ceph since I am not root 2024-09-21T05:37:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:06 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1116031598' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T05:37:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:06 smithi028 ceph-mon[31227]: mgrmap e10: x(active, since 21s) 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 10, 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-21T05:37:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 10... 2024-09-21T05:37:08.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:07 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4228744559' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T05:37:11.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: Active manager daemon x restarted 2024-09-21T05:37:11.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: Activating manager daemon x 2024-09-21T05:37:11.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: mgrmap e11: x(active, starting, since 0.00391765s) 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:37:11.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:11 smithi028 ceph-mon[31227]: Manager daemon x is now available 2024-09-21T05:37:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:12 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:12 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:37:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:12 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:37:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:12 smithi028 ceph-mon[31227]: mgrmap e12: x(active, since 1.00677s) 2024-09-21T05:37:12.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-21T05:37:12.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 12, 2024-09-21T05:37:12.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T05:37:12.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-21T05:37:12.667 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 10 is available 2024-09-21T05:37:12.668 INFO:teuthology.orchestra.run.smithi028.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T05:37:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-mon[31227]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T05:37:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-mon[31227]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:37:13.680+0000 7f8165a0f640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:05:37:13] ENGINE Error in HTTPServer.serve 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Traceback (most recent call last): 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._connections.run(self.expiration_interval) 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._run(expiration_interval) 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:37:14.047 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s = self.context.wrap_socket( 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: return self.sslsocket_class._create( 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self.do_handshake() 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._sslobj.do_handshake() 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:37:14.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:37:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: mgrmap e13: x(active, since 2s) 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: [21/Sep/2024:05:37:13] ENGINE Bus STARTING 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: [21/Sep/2024:05:37:13] ENGINE Serving on https://172.21.15.28:7150 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: [21/Sep/2024:05:37:13] ENGINE Error in HTTPServer.serve 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:37:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:37:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-21T05:37:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:37:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: [21/Sep/2024:05:37:13] ENGINE Serving on http://172.21.15.28:8765 2024-09-21T05:37:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: [21/Sep/2024:05:37:13] ENGINE Bus STARTED 2024-09-21T05:37:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:14 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:14.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUTswrspUJhgjbSrC90N1Olkw5JtEwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTA1MzcxNFoXDTM0\nMDkyMjA1MzcxNFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjI4MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA3rR20KE32jA6423vu3tVa5hk8WMHW+HRuD7F\nj+BbNTAADix8Jfk3sxz5ezsnlaJxtLtnekDcK5/i5JRLfXULndft267mhrtwzD1w\nUQeJQ5Hrz0rIXSX/RT/uD2ozDR2LNxuLxGzqNRLy8x3Ah114t/kGhCMnqgvdxf9j\nmF2vQ27bbZ79wgAMin0Uwk/0AXvioazbB4XBo2Ry58XPLnVRGXZUEC/gnru2HuCb\nGXvdNhjeYskk6wvp6EA8QFcvKCD/jd0+Guv019a9T5jXs7QeKytnig/4KXmYn/GI\nIUd5+dmEc1eiqyYAYC9EX9CkghNmsykw/M8DmlIIHSThpZAozg2RFdQLm7v+tSQ7\nJXvtd1eK1YBhFyxvGGaCHC16mwaVSxU/ibqiRYgbEdwc6Rt0/urNK1LcXhl8nXuK\nGZtERLR+bnu7H2yZ5waG5lRfRgv293CieBft//zf43sBv0ODkUR+Rq0k8efs1sMN\nwUCbeDah6SLb68mecwzBLOAzLaIE2GnI/p9MveGe5zv7VSymrlDC3R6gSoQt/jQH\ncFkSG3zOb7K9U/iytokRgaeaoLK7ri6NBdIyO/+GpacUtHJpz3DPwzAssJOylrFI\n1IAxbV0hQqSduKLop3TQZ1ETT5pPB/N+tE2JFtvaTaniCbIHYvcEv2NtYkBDR3Wt\nTQH0qWsCAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDI4ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPHDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAT\nqkNGd6Y0D3W1PUO8AQx4hjeeOQ3qkHtY+whDP6jLsK7WTwNPzzS+WYPVrQTkksiO\nLW2rsY5F/zMZ1zvVuiICYElb2QP6OjNdnTWidObKYu8mgpP1GaCbH8A3VzRoP0jZ\n4EdbRlx5o6QrhiucX7Yn7KmWkazlcqK2zXrY6Oa0Na8s1SqDRhZUT12zhE40gNpi\nphHrykREiKVwvShi/VgHdzW8q/q+/LlbZEJj+XPmnIGzGztl00LC3Nv4akmMx8lr\n7dNMsqUQShDAHSJXyGUT0zrtLJIFnSxccNzHkaJWkunmV2RbkIBvj0bCf2biGyB9\nE10dYRwIzQdFKi5hfaEUFhoE/pgIVX3DbShb81SX7YpJzUUWmuTVwFvSZe6mDFVS\nBvuNBXfy31LWAgTk4yapHzJKuGMFU6X0DZzIzeRB+bBIYHrRTvYbppgcQH5QSmeX\nIn/1DoQHUHBeRAskPKJ/Og1EWhhgkd1WImh+is5hjGxYtdLK3KXjnq6yEbn1BCo4\nVsF2TuClM0oD+57pmRM23yKZ24LJDPvcEp2SpGiCm1LjBBmtOhevTKxStAl7tBt+\nYL3mx2rPpOH1Nj6Dx/KesP0XUYk8wHr6XKMJrocKvaZkInKbD+UXh3yEnut60v40\nry7PVtvPTJnrzVYbSe2I9Hi4L01rtcTtJMiKR5ipcQ==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEA3rR20KE32jA6423vu3tVa5hk8WMHW+HRuD7Fj+BbNTAADix8\nJfk3sxz5ezsnlaJxtLtnekDcK5/i5JRLfXULndft267mhrtwzD1wUQeJQ5Hrz0rI\nXSX/RT/uD2ozDR2LNxuLxGzqNRLy8x3Ah114t/kGhCMnqgvdxf9jmF2vQ27bbZ79\nwgAMin0Uwk/0AXvioazbB4XBo2Ry58XPLnVRGXZUEC/gnru2HuCbGXvdNhjeYskk\n6wvp6EA8QFcvKCD/jd0+Guv019a9T5jXs7QeKytnig/4KXmYn/GIIUd5+dmEc1ei\nqyYAYC9EX9CkghNmsykw/M8DmlIIHSThpZAozg2RFdQLm7v+tSQ7JXvtd1eK1YBh\nFyxvGGaCHC16mwaVSxU/ibqiRYgbEdwc6Rt0/urNK1LcXhl8nXuKGZtERLR+bnu7\nH2yZ5waG5lRfRgv293CieBft//zf43sBv0ODkUR+Rq0k8efs1sMNwUCbeDah6SLb\n68mecwzBLOAzLaIE2GnI/p9MveGe5zv7VSymrlDC3R6gSoQt/jQHcFkSG3zOb7K9\nU/iytokRgaeaoLK7ri6NBdIyO/+GpacUtHJpz3DPwzAssJOylrFI1IAxbV0hQqSd\nuKLop3TQZ1ETT5pPB/N+tE2JFtvaTaniCbIHYvcEv2NtYkBDR3WtTQH0qWsCAwEA\nAQKCAgAIcHdGLa3Tp1s77hkVFjXk92fv28xWF4cef8Z1zs82A0UlRFtKBxkswO7V\nVCcKXRcI/ccMk3OkdZEgD1yS+wH/L8UUO5sBxQ2ft75PW8VAWcHBX7ibCwsS4TrQ\nu7oGh7WUV4di4Ku5aifpvZljp8+gzR+Jl1lz8LItRCHOjtJK8GMBUvKTzPOstdO5\nD1WUTtEGRmogbVmqqu5NySmyPYDewaSq3l2ZNkzO+7z06MQ3vhaDUNME0zVAd1qx\nJ3yifSmiAx3M4F8tZLZU01ptNWilQyZPzY4eaDy35uu9Mxu0s6D4y/vel3iRLIlf\nRiVNxpVwOaHhGqksymoDTLf+UamyHyGEQSf7nMJOYR5lFZ1oOi4ZSeIxwIJ4uYYt\nHgNTfppO7qOI0xYReIdGM1PBL7t+uL+M2FLgZ8451W/0C6vIOpDOV5Mv42BEkBtg\njLcny7J89bZ81XmzNn2pP+oP87jQMUoo5uXzA9bCnGHUBFHsdEZCccYEPAGodBvI\nxVmpDars6jjmAa9cvoDQlQYXRS5P77+1Bj1kaU6lZMZv7qEj/vEiv1N6xHQVDm4i\n0a6fRacJIwTh4W3zsc95OkjXw4hcXtEW6QiHDWN9DD9wB9CabeJDvXhU5xsk4nfI\nlMUE0sIxRz206xuVmryDvWebEBFqgdSZdTw58Ws77ZqkEFwo9QKCAQEA/W3O9rRJ\nSandZj715mB+7EkKKMel1NVJXZ8ToNXFOmachRkcdQMnTHMj2Lp8HxXdE6AeAxhX\nRc+/by+yiFjMrBu6M+WtHhp23DaobLHPR/gHNuBYZ9RJOUBI+47QIAHRRU6KH3Cs\nsU4e5xuJoMJ6ZUSHnU8X/PBDMRQBugceefqVJEbB3FtEn3T3XWHgmL2PmXyB6vgy\n7MRzTRwKID22GB9WjZ1G4g7yQoZdxb1k7JSLYEwyAK/VKamfy3oMg5IaSLla9hA3\noIJiCSzx1V8B4yXnhoKviC0uCGetGzXl3GedcfzuFWe7MX+six0wjeOwp7SVtyWS\nh2HMOHvwdwQrfwKCAQEA4PbcapaV8HVm4H0eUTWTFP8n9jCy+7uysCUX44vjkO51\nkKs/53f+RKZ2iFzbpNZz0NwtBZObWMo7HOtjCnwnBv2KVtQ2ndqEEpx0HZ5273ds\nUElpchgYxhxBHpKlJjYL2nQcTyZWcxmLKL9OVp18YUte8Pa9108MRJZzpvpiaVf6\nEbrobuUPTkUJz5VaeG5kB/jPyoX0/MkuMHwaS+RcUE/iT2qRjbaGvD3N4WgJL1CL\nYAHAqjOVgPWahkxCJ2u9sacBZEcJKwCJnz7aPX3RN9X5/eK3HEYRGDoG5Y4VSllr\nSSzR1b1mWkzEfCz2cxnlaYk9U6n6H/FfHWKrCc7oFQKCAQATjCe9kbRhcS+5ljuj\nVGgV06Br+tJIs1bgWMfLzRFUatvBzGkMgXzDgdIPAntWbnpJ7DzEYllA8kBC1DwC\nFk9bKxxWqm2K9+GZJMIi+CgT1gbAbdeD0rT/LH00ohRdk6ZPoTo4KJm5Z1ws2Uvp\nUXbdXsbTFGm2QdX88B0d+zSXlgNw8+A6HfQSFJSL6GPJhD2SSi8/L3vGFXULSpFh\nYUEtZSVo8OupvhtjlDeha/Vf5ilF+TqWIPnhJ8tCh5x1fYx3dYyVlDSmNchYCtZw\nbc6YdbEBCA2ABHQbg9uvUJ31PSzAwRCgKAshS0bEyDCZkSlsFhrCMKy3soxtAupO\nU80DAoIBAQClkqt5u6lRmGcGdLAany7Itl9IIGGrS2Nhp3tkm3H7GrbmqlPXXgaD\nqmz1QjraoIgPQe3U++XLPNSzHUAA2I205GisLXahSdIJDKOdpSLubtwGX3G/aqLZ\nNLRs9wucCw1AVFL9oj7gaf11GRBkSXbg+SNAgKoKzQA/fuhSDljeJcVExWcLL4XL\nWp607LzwhHcVMvhkm6FLaBqZzs1f7AxRg+oR/2Jvhda2t1pPO03/R+281K1j16GC\nj8GSdJAACZlNJpisoGfMAZJI5W6k8XRJ/zAGv5V7tsogJZ0r2R4sYZHIArfR85tg\nfL3kkwc0gevKeBrt993DItiZfSRbjLBdAoIBAHzf0a86B7qgjfmvO0vMX3VZwsrM\nP9xMPPm61seIIpJkav8x0QsyWjgwITZRAVZfFe78HyoC3rN+RHuW+nku3mBUuzcV\nnQEehX8r6JfemwLo/b2tywls7f3sGtQxiF8LexV0hJdi8J2O3xZeDbDDMZLhPgBr\n3Dby7IGmtblLYYGoT691mXZJ2F2B+NWIrIX6QBucfxUkSkpzfvevMBhHL9GQQpot\njmvsrizsCM8npkpE+5/FRRgqq/uNyqlZCMqO4lMQZAvdvtHYDsD4Ze4xR/V8QYlC\nGrO3DisFhvBOddAOQN6yxSJ7no20oM7pnnoc7FJ8Rt2F0V7d0Ht1copLw34=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T05:37:16.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T05:37:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:16 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:16.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:16 smithi028 ceph-mon[31227]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:16.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:16 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:17.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T05:37:17.392 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-09-21T05:37:18.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:17 smithi028 ceph-mon[31227]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:18.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:17 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:19.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ZqC.MNUHTkNRpcDcOtO6Wu80VobdqdLkpegTnYOrx2gPKV2lKFXQC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726897038, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T05:37:19.036 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-09-21T05:37:19.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:19 smithi028 ceph-mon[31227]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:19.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:19 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:20.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:20 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1364869642' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T05:37:20.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T05:37:20.311 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T05:37:20.311 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-21T05:37:20.319 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T05:37:20.319 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-21T05:37:20.319 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-21T05:37:20.319 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T05:37:20.320 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: Password: pfpt1g86j6 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:20.321 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config directory 2024-09-21T05:37:21.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-09-21T05:37:21.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.584 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-09-21T05:37:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T05:37:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:21.584 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-09-21T05:37:21.623 INFO:tasks.cephadm:Fetching config... 2024-09-21T05:37:21.623 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:37:21.623 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T05:37:21.641 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T05:37:21.641 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:37:21.641 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T05:37:21.696 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T05:37:21.696 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:37:21.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T05:37:21.763 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T05:37:21.763 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:37:21.763 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T05:37:21.819 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T05:37:21.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T05:37:21.915 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:21.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T05:37:21.970 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:21.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T05:37:22.025 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkEgAgebUOQQB+/by2mx4/s1wPjWQo4IHzjey4eQOsHLvAyN2dC+ii51jGDUywAlKKPmeCunW0pgbsvXDs2BKiHiufCcrzU6YUOA/nN8IdQM1DSSA3c8r7H8Vhhra5sPWdOj4PzQqB5nPhk6Y3l/H0c2V9qb2nyOjJ2hxn4lE/gDeN6zmEcurHnGHz6VJFLcc5kP0OBuCixgeYMwl7dJ9U3V3JJ+vpljh6TtsKVDxWJKKqzbzFQWUDrfgPK/XfJthDGRmadTM0YQM6MnNCiLLg3pDS97PMDfZO8y6Ahnce+36BtW93M8Lxkz9MOGtw3QE2gk9juXA6gwsakQ1lBqunz5gi4j/Vo1vPlUol2qL1HFr2rB6iTHJQTAM6K1DHQckBa+WnnH23FIQqQ06wNzYH+UzwqL+OvMUWefMpoq8e3fIYPFW4C9EeKSdGfdMjyhwsqUYsX9S6vfhN5ojq10jyal88I1s3DTBxqV85wUbM5z201YdT8kldrG/JcOSeML0= ceph-42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:22.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T05:37:22.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:22 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3241667786' entity='client.admin' 2024-09-21T05:37:22.295 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:24.102 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T05:37:24.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1621721337' entity='client.admin' 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:24.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:23 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:25.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:24 smithi028 ceph-mon[31227]: mgrmap e14: x(active, since 12s) 2024-09-21T05:37:25.970 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-09-21T05:37:25.970 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:37:25.970 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-09-21T05:37:25.989 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:37:25.989 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:26.047 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-09-21T05:37:26.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch host add smithi129 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:26.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:26.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:26 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:27 smithi028 ceph-mon[31227]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:29.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:28 smithi028 ceph-mon[31227]: Deploying cephadm binary to smithi129 2024-09-21T05:37:30.802 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-09-21T05:37:31.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T05:37:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:32.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:31 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:32.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:31 smithi028 ceph-mon[31227]: Added host smithi129 2024-09-21T05:37:32.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:31 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:32.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:31 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:32.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:31 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:32.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:32.350 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-09-21T05:37:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:32 smithi028 ceph-mon[31227]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:32 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:32.964 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-09-21T05:37:32.964 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:37:32.964 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-09-21T05:37:32.983 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:37:32.983 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:33.043 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-09-21T05:37:33.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch host add smithi181 2024-09-21T05:37:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:33 smithi028 ceph-mon[31227]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:37:35.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:34 smithi028 ceph-mon[31227]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:35 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: Deploying cephadm binary to smithi181 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:37.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:36 smithi028 ceph-mon[31227]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:37.830 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-09-21T05:37:38.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T05:37:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:39.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:39.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:39.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: Added host smithi181 2024-09-21T05:37:39.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:39.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:39.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:38 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:39.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:37:39.395 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-09-21T05:37:39.952 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T05:37:39.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-21T05:37:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:37:40.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:40 smithi028 ceph-mon[31227]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:40.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:40 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:40.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:40 smithi028 ceph-mon[31227]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:37:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-21T05:37:41.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:41 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T05:37:41.705 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-09-21T05:37:41.705 INFO:tasks.cephadm:Adding mon.b on smithi129 2024-09-21T05:37:41.705 INFO:tasks.cephadm:Adding mon.c on smithi181 2024-09-21T05:37:41.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c' 2024-09-21T05:37:41.965 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T05:37:42.011 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T05:37:42.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:42.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T05:37:42.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:42.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:42.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:42.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:42 smithi028 ceph-mon[31227]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:37:42.706 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-09-21T05:37:43.303 DEBUG:teuthology.orchestra.run.smithi129:mon.b> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.b.service 2024-09-21T05:37:43.306 DEBUG:teuthology.orchestra.run.smithi181:mon.c> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.c.service 2024-09-21T05:37:43.308 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T05:37:43.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T05:37:43.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c;count:3 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:43.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:43.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:43.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:43.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:43.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:43 smithi028 ceph-mon[31227]: Deploying daemon mon.c on smithi181 2024-09-21T05:37:44.316 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:45.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:44 smithi028 ceph-mon[31227]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:46.196 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:37:46.196 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"42500114-77db-11ef-baf5-efdc52797490","modified":"2024-09-21T05:36:09.260492Z","created":"2024-09-21T05:36:09.260492Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T05:37:46.196 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-09-21T05:37:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:46 smithi181 systemd[1]: Starting Ceph mon.c for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:37:47.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:46 smithi028 ceph-mon[31227]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:46 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/863884718' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:47.064 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:46 smithi181 podman[34800]: 2024-09-21 05:37:46.955401638 +0000 UTC m=+0.115700812 container create 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T05:37:47.064 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:46 smithi181 podman[34800]: 2024-09-21 05:37:46.865208026 +0000 UTC m=+0.025507210 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:37:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 podman[34800]: 2024-09-21 05:37:47.166103177 +0000 UTC m=+0.326402352 container init 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T05:37:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 podman[34800]: 2024-09-21 05:37:47.169279356 +0000 UTC m=+0.329578531 container start 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T05:37:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T05:37:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T05:37:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pidfile_write: ignore empty --pid-file 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: load: jerasure load: lrc 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: RocksDB version: 7.9.2 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Git sha 0 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: DB SUMMARY 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: DB Session ID: OUBOLKH45I5TS610W01O 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: CURRENT file: CURRENT 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.error_if_exists: 0 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.create_if_missing: 0 2024-09-21T05:37:47.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.paranoid_checks: 1 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.env: 0x55ae65668d60 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.info_log: 0x55ae6b8d3360 2024-09-21T05:37:47.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T05:37:47.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.statistics: (nil) 2024-09-21T05:37:47.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.use_fsync: 0 2024-09-21T05:37:47.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_log_file_size: 0 2024-09-21T05:37:47.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_fallocate: 1 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.use_direct_reads: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.db_log_dir: 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.wal_dir: 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T05:37:47.548 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.write_buffer_manager: 0x55ae6b8d7d60 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T05:37:47.549 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.unordered_write: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.row_cache: None 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.wal_filter: None 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.two_write_queues: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.wal_compression: 0 2024-09-21T05:37:47.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.atomic_flush: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.log_readahead_size: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_background_jobs: 2 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_background_compactions: -1 2024-09-21T05:37:47.551 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_subcompactions: 1 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_open_files: -1 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_background_flushes: -1 2024-09-21T05:37:47.552 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Compression algorithms supported: 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kZSTD supported: 0 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kXpressCompression supported: 0 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kBZip2Compression supported: 0 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kLZ4Compression supported: 1 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kZlibCompression supported: 1 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: kSnappyCompression supported: 1 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T05:37:47.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.merge_operator: 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_filter: None 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ae6b8d2fa0) 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks: 1 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_top_level_index_and_filter: 1 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_type: 0 2024-09-21T05:37:47.554 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_index_type: 0 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_shortening: 1 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: checksum: 4 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: no_block_cache: 0 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache: 0x55ae67897610 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_name: BinnedLRUCache 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_options: 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: capacity : 536870912 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_shard_bits : 4 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: strict_capacity_limit : 0 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: high_pri_pool_ratio: 0.000 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_compressed: (nil) 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: persistent_cache: (nil) 2024-09-21T05:37:47.555 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size: 4096 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size_deviation: 10 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_restart_interval: 16 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_block_restart_interval: 1 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: metadata_block_size: 4096 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: partition_filters: 0 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: use_delta_encoding: 1 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: filter_policy: bloomfilter 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: whole_key_filtering: 1 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: verify_compression: 0 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: read_amp_bytes_per_bit: 0 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: format_version: 5 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_index_compression: 1 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_align: 0 2024-09-21T05:37:47.556 INFO:journalctl@ceph.mon.c.smithi181.stdout: max_auto_readahead_size: 262144 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout: prepopulate_block_cache: 0 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout: initial_auto_readahead_size: 8192 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression: NoCompression 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.num_levels: 7 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T05:37:47.557 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:37:47.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T05:37:47.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T05:37:47.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.inplace_update_support: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.bloom_locality: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.max_successive_merges: 0 2024-09-21T05:37:47.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.ttl: 2592000 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enable_blob_files: false 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.min_blob_size: 0 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T05:37:47.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 71e55a0d-1c4c-4b5d-bf72-d45a55f3738d 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897067207682, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897067208122, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726897067, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "71e55a0d-1c4c-4b5d-bf72-d45a55f3738d", "db_session_id": "OUBOLKH45I5TS610W01O", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897067208217, "job": 1, "event": "recovery_finished"} 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T05:37:47.563 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ae6b8f6a80 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: DB pointer 0x55ae6c69c000 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** DB Stats ** 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2024-09-21T05:37:47.564 INFO:journalctl@ceph.mon.c.smithi181.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:37:47.565 INFO:journalctl@ceph.mon.c.smithi181.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache BinnedLRUCache@0x55ae67897610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: using public_addr v2:172.21.15.181:0/0 -> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-21T05:37:47.566 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: starting mon.c rank -1 at public addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] at bind addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(???) e0 preinit fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: e1 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: btime 2024-09-21T05:36:12:178942+0000 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: legacy client fscid: -1 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout: No filesystems configured 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T05:37:47.567 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:47.568 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.569 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Deploying cephadm binary to smithi181 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Added host smithi181 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T05:37:47.570 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:47.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c;count:3 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: Deploying daemon mon.c on smithi181 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:47.573 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/863884718' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:47.573 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 ceph-mon[34819]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T05:37:47.573 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 bash[34800]: 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 2024-09-21T05:37:47.573 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:47 smithi181 systemd[1]: Started Ceph mon.c for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:37:47.830 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T05:37:47.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T05:37:48.090 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: mon.a calling monitor election 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: mon.c calling monitor election 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: monmap epoch 2 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: last_changed 2024-09-21T05:37:47.236314+0000 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: min_mon_release 19 (squid) 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: election_strategy: 1 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:37:52.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: fsmap 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: mgrmap e14: x(active, since 41s) 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: overall HEALTH_OK 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:52.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:52 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:37:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: mon.a calling monitor election 2024-09-21T05:37:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: mon.c calling monitor election 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: monmap epoch 2 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: last_changed 2024-09-21T05:37:47.236314+0000 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: min_mon_release 19 (squid) 2024-09-21T05:37:52.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: election_strategy: 1 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: fsmap 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: mgrmap e14: x(active, since 41s) 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: overall HEALTH_OK 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:52.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:52 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:37:53.240 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"42500114-77db-11ef-baf5-efdc52797490","modified":"2024-09-21T05:37:47.236314Z","created":"2024-09-21T05:36:09.260492Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T05:37:53.241 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-09-21T05:37:53.540 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:53 smithi181 ceph-mon[34819]: Deploying daemon mon.b on smithi129 2024-09-21T05:37:53.540 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:53 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:53.541 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:53 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/1802603586' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:53.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:53 smithi028 ceph-mon[31227]: Deploying daemon mon.b on smithi129 2024-09-21T05:37:53.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:53 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:53.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:53 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/1802603586' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:54.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:37:54 smithi181 ceph-mon[34819]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:54.811 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T05:37:54.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T05:37:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:37:54 smithi028 ceph-mon[31227]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:55.064 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:37:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:37:55.761 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"42500114-77db-11ef-baf5-efdc52797490","modified":"2024-09-21T05:37:47.236314Z","created":"2024-09-21T05:36:09.260492Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T05:37:55.761 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-09-21T05:37:55.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 podman[34866]: 2024-09-21 05:37:55.9145293 +0000 UTC m=+0.340761223 container start d1186cbf6d5bdfbe3157847d5fd643e655c9d96e8a75aa1710fb35c99c0e6a8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T05:37:56.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pidfile_write: ignore empty --pid-file 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: load: jerasure load: lrc 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: RocksDB version: 7.9.2 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Git sha 0 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: DB SUMMARY 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: DB Session ID: H4SYVL5SDK4AEW13GLHS 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: CURRENT file: CURRENT 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T05:37:56.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.error_if_exists: 0 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.create_if_missing: 0 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.paranoid_checks: 1 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.env: 0x563b0bbbdd60 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.info_log: 0x563b11bf3360 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.statistics: (nil) 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.use_fsync: 0 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_log_file_size: 0 2024-09-21T05:37:56.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_fallocate: 1 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.use_direct_reads: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.db_log_dir: 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.wal_dir: 2024-09-21T05:37:56.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.write_buffer_manager: 0x563b11bf7d60 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T05:37:56.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.unordered_write: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.row_cache: None 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.wal_filter: None 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T05:37:56.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.two_write_queues: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.wal_compression: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.atomic_flush: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.log_readahead_size: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T05:37:56.371 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_background_jobs: 2 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_background_compactions: -1 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_subcompactions: 1 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T05:37:56.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_open_files: -1 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_background_flushes: -1 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Compression algorithms supported: 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kZSTD supported: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kXpressCompression supported: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kBZip2Compression supported: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kLZ4Compression supported: 1 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kZlibCompression supported: 1 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T05:37:56.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: kSnappyCompression supported: 1 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.merge_operator: 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_filter: None 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T05:37:56.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563b11bf2fa0) 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_type: 0 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_index_type: 0 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_shortening: 1 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: checksum: 4 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: no_block_cache: 0 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache: 0x563b0dbb7610 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-09-21T05:37:56.375 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_options: 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: capacity : 536870912 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_shard_bits : 4 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: strict_capacity_limit : 0 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_compressed: (nil) 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: persistent_cache: (nil) 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size: 4096 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size_deviation: 10 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_restart_interval: 16 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_block_restart_interval: 1 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_block_size: 4096 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: partition_filters: 0 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: use_delta_encoding: 1 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: filter_policy: bloomfilter 2024-09-21T05:37:56.376 INFO:journalctl@ceph.mon.b.smithi129.stdout: whole_key_filtering: 1 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: verify_compression: 0 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: format_version: 5 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_index_compression: 1 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_align: 0 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_auto_readahead_size: 262144 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: prepopulate_block_cache: 0 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: initial_auto_readahead_size: 8192 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression: NoCompression 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T05:37:56.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.num_levels: 7 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:37:56.378 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T05:37:56.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T05:37:56.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.inplace_update_support: 0 2024-09-21T05:37:56.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.bloom_locality: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.max_successive_merges: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.ttl: 2592000 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enable_blob_files: false 2024-09-21T05:37:56.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.min_blob_size: 0 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8aa37cf4-8d7f-4342-81e2-8afd9ee6d71e 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897075951353, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T05:37:56.383 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897075951787, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726897075, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8aa37cf4-8d7f-4342-81e2-8afd9ee6d71e", "db_session_id": "H4SYVL5SDK4AEW13GLHS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726897075951880, "job": 1, "event": "recovery_finished"} 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563b11c16a80 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: DB pointer 0x563b129bc000 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** DB Stats ** 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T05:37:56.384 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.385 INFO:journalctl@ceph.mon.b.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache BinnedLRUCache@0x563b0dbb7610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T05:37:56.386 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: using public_addr v2:172.21.15.129:0/0 -> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: starting mon.b rank -1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(???) e0 preinit fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: e1 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: btime 2024-09-21T05:36:12:178942+0000 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: legacy client fscid: -1 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T05:37:56.387 INFO:journalctl@ceph.mon.b.smithi129.stdout: No filesystems configured 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Deploying cephadm binary to smithi181 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:56.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Added host smithi181 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4217512261' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.390 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c;count:3 2024-09-21T05:37:56.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Deploying daemon mon.c on smithi181 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.392 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/863884718' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.a calling monitor election 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.c calling monitor election 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:37:56.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: monmap epoch 2 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: last_changed 2024-09-21T05:37:47.236314+0000 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: min_mon_release 19 (squid) 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: election_strategy: 1 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: fsmap 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mgrmap e14: x(active, since 41s) 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: overall HEALTH_OK 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: Deploying daemon mon.b on smithi129 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/1802603586' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 ceph-mon[34885]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 bash[34866]: d1186cbf6d5bdfbe3157847d5fd643e655c9d96e8a75aa1710fb35c99c0e6a8e 2024-09-21T05:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:37:55 smithi129 systemd[1]: Started Ceph mon.b for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:37:57.301 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T05:37:57.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T05:37:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: mon.c calling monitor election 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: mon.a calling monitor election 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: monmap epoch 3 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: min_mon_release 19 (squid) 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: election_strategy: 1 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:01.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: fsmap 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: mgrmap e14: x(active, since 49s) 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: overall HEALTH_OK 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.296 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:01 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:01.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: mon.c calling monitor election 2024-09-21T05:38:01.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: mon.a calling monitor election 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: monmap epoch 3 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: min_mon_release 19 (squid) 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: election_strategy: 1 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: fsmap 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: mgrmap e14: x(active, since 49s) 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: overall HEALTH_OK 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:01 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:38:01.459 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"42500114-77db-11ef-baf5-efdc52797490","modified":"2024-09-21T05:37:55.978962Z","created":"2024-09-21T05:36:09.260492Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T05:38:01.459 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/2492545227' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:02 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/2492545227' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:02 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:02.777 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T05:38:02.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T05:38:03.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: mon.c calling monitor election 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: mon.a calling monitor election 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: monmap epoch 3 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: min_mon_release 19 (squid) 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: election_strategy: 1 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: fsmap 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:03.359 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: mgrmap e14: x(active, since 49s) 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: overall HEALTH_OK 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/2492545227' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:03.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.361 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:02 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:03.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mon.b calling monitor election 2024-09-21T05:38:03.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mon.b calling monitor election 2024-09-21T05:38:03.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mon.c calling monitor election 2024-09-21T05:38:03.955 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mon.a calling monitor election 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: monmap epoch 3 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: min_mon_release 19 (squid) 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: election_strategy: 1 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: fsmap 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: mgrmap e14: x(active, since 51s) 2024-09-21T05:38:03.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:03 smithi129 ceph-mon[34885]: overall HEALTH_OK 2024-09-21T05:38:04.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mon.b calling monitor election 2024-09-21T05:38:04.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mon.b calling monitor election 2024-09-21T05:38:04.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mon.c calling monitor election 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mon.a calling monitor election 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: monmap epoch 3 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: min_mon_release 19 (squid) 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: election_strategy: 1 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: fsmap 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:04.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: mgrmap e14: x(active, since 51s) 2024-09-21T05:38:04.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:03 smithi028 ceph-mon[31227]: overall HEALTH_OK 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mon.b calling monitor election 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mon.b calling monitor election 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mon.c calling monitor election 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mon.a calling monitor election 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: monmap epoch 3 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: last_changed 2024-09-21T05:37:55.978962+0000 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: created 2024-09-21T05:36:09.260492+0000 2024-09-21T05:38:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: min_mon_release 19 (squid) 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: election_strategy: 1 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: 1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: fsmap 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: mgrmap e14: x(active, since 51s) 2024-09-21T05:38:04.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:03 smithi181 ceph-mon[34819]: overall HEALTH_OK 2024-09-21T05:38:04.531 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:04.531 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-21T05:38:04.531 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:38:04.531 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:05.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:04 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2439347086' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:05.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:05.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:04 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2439347086' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:05.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:04 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2439347086' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:05.188 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T05:38:05.188 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:38:05.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T05:38:05.217 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:38:05.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:05.284 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:38:05.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T05:38:05.313 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:38:05.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:05.378 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:38:05.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T05:38:05.405 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:38:05.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:05.470 INFO:tasks.cephadm:enabling module snap_schedule 2024-09-21T05:38:05.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-09-21T05:38:05.721 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: Reconfiguring mon.b (monmap changed)... 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: Reconfiguring daemon mon.b on smithi129 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:05.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:05 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: Reconfiguring mon.b (monmap changed)... 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: Reconfiguring daemon mon.b on smithi129 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:06.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:05 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: Reconfiguring mon.b (monmap changed)... 2024-09-21T05:38:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: Reconfiguring daemon mon.b on smithi129 2024-09-21T05:38:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T05:38:06.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T05:38:06.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:05 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: Reconfiguring mon.c (monmap changed)... 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: Reconfiguring daemon mon.c on smithi181 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:06.663 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:07.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:07.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:07.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:06 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: Reconfiguring mon.c (monmap changed)... 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: Reconfiguring daemon mon.c on smithi181 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:07.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:07.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:06 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: Reconfiguring mon.c (monmap changed)... 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: Reconfiguring daemon mon.c on smithi181 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:07.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:07.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:07.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='mgr.14152 172.21.15.28:0/3905402800' entity='mgr.x' 2024-09-21T05:38:07.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:06 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T05:38:07.741 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setuser ceph since I am not root 2024-09-21T05:38:07.741 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ignoring --setgroup ceph since I am not root 2024-09-21T05:38:08.030 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-09-21T05:38:08.030 INFO:tasks.cephadm:Adding mgr.y on smithi129 2024-09-21T05:38:08.030 INFO:tasks.cephadm:Adding mgr.z on smithi181 2024-09-21T05:38:08.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch apply mgr '3;smithi028=x;smithi129=y;smithi181=z' 2024-09-21T05:38:08.282 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:38:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:08 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T05:38:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:08 smithi129 ceph-mon[34885]: mgrmap e15: x(active, since 56s) 2024-09-21T05:38:08.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:08 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T05:38:08.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:08 smithi181 ceph-mon[34819]: mgrmap e15: x(active, since 56s) 2024-09-21T05:38:08.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:08 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1919564510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T05:38:08.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:08 smithi028 ceph-mon[31227]: mgrmap e15: x(active, since 56s) 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: Active manager daemon x restarted 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: Activating manager daemon x 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: mgrmap e16: x(active, starting, since 0.0048086s) 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:38:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:38:13.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:38:13.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:12 smithi181 ceph-mon[34819]: Manager daemon x is now available 2024-09-21T05:38:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: Active manager daemon x restarted 2024-09-21T05:38:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: Activating manager daemon x 2024-09-21T05:38:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T05:38:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: mgrmap e16: x(active, starting, since 0.0048086s) 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:38:13.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:12 smithi028 ceph-mon[31227]: Manager daemon x is now available 2024-09-21T05:38:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: Active manager daemon x restarted 2024-09-21T05:38:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: Activating manager daemon x 2024-09-21T05:38:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T05:38:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: mgrmap e16: x(active, starting, since 0.0048086s) 2024-09-21T05:38:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T05:38:13.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:12 smithi129 ceph-mon[34885]: Manager daemon x is now available 2024-09-21T05:38:13.644 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-09-21T05:38:13.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:13.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:38:13.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:38:13.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:13 smithi181 ceph-mon[34819]: mgrmap e17: x(active, since 1.01054s) 2024-09-21T05:38:13.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:13.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:13.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:38:13.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:38:13.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:13 smithi129 ceph-mon[34885]: mgrmap e17: x(active, since 1.01054s) 2024-09-21T05:38:13.963 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:14.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:14.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T05:38:14.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T05:38:14.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:13 smithi028 ceph-mon[31227]: mgrmap e17: x(active, since 1.01054s) 2024-09-21T05:38:14.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:14.251 DEBUG:teuthology.orchestra.run.smithi129:mgr.y> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.y.service 2024-09-21T05:38:14.254 DEBUG:teuthology.orchestra.run.smithi181:mgr.z> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.z.service 2024-09-21T05:38:14.257 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-21T05:38:14.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-21T05:38:14.260 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-21T05:38:14.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-21T05:38:14.306 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-21T05:38:14.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-21T05:38:14.350 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-21T05:38:14.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-21T05:38:14.395 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-21T05:38:14.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-21T05:38:14.440 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-21T05:38:14.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-21T05:38:14.486 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-21T05:38:14.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-21T05:38:14.551 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-21T05:38:14.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-21T05:38:14.597 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-21T05:38:14.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-09-21T05:38:14.643 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-09-21T05:38:14.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_merge_size 5 2024-09-21T05:38:14.689 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-09-21T05:38:14.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_bits 3 2024-09-21T05:38:14.727 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:14.736 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-09-21T05:38:14.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_size 100 2024-09-21T05:38:14.787 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-21T05:38:14.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-21T05:38:14.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:14.801 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-21T05:38:14.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-21T05:38:14.850 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-21T05:38:14.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-21T05:38:14.900 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-21T05:38:14.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:38:14.683+0000 7f3f77729640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:05:38:14] ENGINE Error in HTTPServer.serve 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: Traceback (most recent call last): 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._connections.run(self.expiration_interval) 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._run(expiration_interval) 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:38:14.919 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: s = self.context.wrap_socket( 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: return self.sslsocket_class._create( 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self.do_handshake() 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: self._sslobj.do_handshake() 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:38:14.920 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:38:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:38:14.923 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-21T05:38:14.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-21T05:38:14.985 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T05:38:14.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-21T05:38:15.040 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T05:38:15.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-21T05:38:15.092 INFO:tasks.cephadm:setting config [mgr] client_mount_timeout = 30 2024-09-21T05:38:15.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr client_mount_timeout 30 2024-09-21T05:38:15.148 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-09-21T05:38:15.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr debug_cephsqlite 20 2024-09-21T05:38:15.193 INFO:tasks.cephadm:setting config [mgr] debug_client = 20 2024-09-21T05:38:15.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr debug_client 20 2024-09-21T05:38:15.266 INFO:tasks.cephadm:setting config [mgr] debug_mgr = 20 2024-09-21T05:38:15.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr debug_mgr 20 2024-09-21T05:38:15.320 INFO:tasks.cephadm:setting config [mgr] debug_ms = 1 2024-09-21T05:38:15.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr debug_ms 1 2024-09-21T05:38:15.370 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-09-21T05:38:15.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-09-21T05:38:15.422 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-09-21T05:38:15.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-09-21T05:38:15.480 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-09-21T05:38:15.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: Saving service mgr spec with placement smithi028=x;smithi129=y;smithi181=z;count:3 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:15.544 INFO:tasks.cephadm:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T05:38:15.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: Saving service mgr spec with placement smithi028=x;smithi129=y;smithi181=z;count:3 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.594 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-21T05:38:15.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-21T05:38:15.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: Saving service mgr spec with placement smithi028=x;smithi129=y;smithi181=z;count:3 2024-09-21T05:38:15.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:15.642 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-21T05:38:15.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-21T05:38:15.697 DEBUG:tasks.cephadm:set 30 configs 2024-09-21T05:38:15.697 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi028') 2024-09-21T05:38:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: [21/Sep/2024:05:38:14] ENGINE Bus STARTING 2024-09-21T05:38:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: [21/Sep/2024:05:38:14] ENGINE Serving on https://172.21.15.28:7150 2024-09-21T05:38:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: [21/Sep/2024:05:38:14] ENGINE Error in HTTPServer.serve 2024-09-21T05:38:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-21T05:38:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:38:16.562 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: [21/Sep/2024:05:38:14] ENGINE Serving on http://172.21.15.28:8765 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: [21/Sep/2024:05:38:14] ENGINE Bus STARTED 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: mgrmap e18: x(active, since 2s) 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:16.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:16 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: [21/Sep/2024:05:38:14] ENGINE Bus STARTING 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: [21/Sep/2024:05:38:14] ENGINE Serving on https://172.21.15.28:7150 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: [21/Sep/2024:05:38:14] ENGINE Error in HTTPServer.serve 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._connections.run(self.expiration_interval) 2024-09-21T05:38:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._run(expiration_interval) 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: s = self.context.wrap_socket( 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.sslsocket_class._create( 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.do_handshake() 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._sslobj.do_handshake() 2024-09-21T05:38:16.615 INFO:journalctl@ceph.mon.b.smithi129.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: [21/Sep/2024:05:38:14] ENGINE Serving on http://172.21.15.28:8765 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: [21/Sep/2024:05:38:14] ENGINE Bus STARTED 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: mgrmap e18: x(active, since 2s) 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:16.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:16 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: [21/Sep/2024:05:38:14] ENGINE Bus STARTING 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: [21/Sep/2024:05:38:14] ENGINE Serving on https://172.21.15.28:7150 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: [21/Sep/2024:05:38:14] ENGINE Error in HTTPServer.serve 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: Traceback (most recent call last): 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: self._connections.run(self.expiration_interval) 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: self._run(expiration_interval) 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T05:38:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: s = self.context.wrap_socket( 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: return self.sslsocket_class._create( 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: self.do_handshake() 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: self._sslobj.do_handshake() 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: [21/Sep/2024:05:38:14] ENGINE Serving on http://172.21.15.28:8765 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: [21/Sep/2024:05:38:14] ENGINE Bus STARTED 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: mgrmap e18: x(active, since 2s) 2024-09-21T05:38:16.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:16.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:16.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:16 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.497+0000 7fde82ffd640 1 -- 172.21.15.28:0/4237558429 <== mon.2 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v12)=0 v12) ==== 109+0+0 (secure 0 0 0) 0x7fde8801e120 con 0x7fde8c075c70 2024-09-21T05:38:17.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fde7803fb70 msgr2=0x7fde78042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:17.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fde7803fb70 0x7fde78042080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fde7c000c00 tx=0x7fde7c005d50 comp rx=0 tx=0).stop 2024-09-21T05:38:17.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde8c075c70 msgr2=0x7fde8c19bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:17.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde8c075c70 0x7fde8c19bf30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d6d0 tx=0x7fde8800db90 comp rx=0 tx=0).stop 2024-09-21T05:38:17.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 shutdown_connections 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fde7803fb70 0x7fde78042080 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c10b7a0 0x7fde8c19d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde8c077060 0x7fde8c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 --2- 172.21.15.28:0/4237558429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde8c075c70 0x7fde8c19bf30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 >> 172.21.15.28:0/4237558429 conn(0x7fde8c0ff0f0 msgr2=0x7fde8c107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 shutdown_connections 2024-09-21T05:38:17.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:17.499+0000 7fde94103640 1 -- 172.21.15.28:0/4237558429 wait complete. 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1899655243' entity='client.admin' 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:38:17.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:17 smithi181 ceph-mon[34819]: Deploying daemon mgr.z on smithi181 2024-09-21T05:38:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1899655243' entity='client.admin' 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:17.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:17.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:17.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:17 smithi028 ceph-mon[31227]: Deploying daemon mgr.z on smithi181 2024-09-21T05:38:17.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1899655243' entity='client.admin' 2024-09-21T05:38:17.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"} : dispatch 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi181:/etc/ceph/ceph.conf 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Updating smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:17.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:17.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:17.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:17.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:17.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:17 smithi129 ceph-mon[34885]: Deploying daemon mgr.z on smithi181 2024-09-21T05:38:17.956 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:18.625 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:18 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:18.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:18 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:18.809 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi028') 2024-09-21T05:38:18.810 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-21T05:38:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.856 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:18 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:18.986 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:18 smithi181 systemd[1]: Starting Ceph mgr.z for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:38:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:19.259 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:18 smithi181 podman[36774]: 2024-09-21 05:38:18.98435483 +0000 UTC m=+0.100255451 container create 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T05:38:19.259 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:18 smithi181 podman[36774]: 2024-09-21 05:38:18.899920755 +0000 UTC m=+0.015821368 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:38:19.259 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:19 smithi181 podman[36774]: 2024-09-21 05:38:19.22018836 +0000 UTC m=+0.336088983 container init 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T05:38:19.259 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:19 smithi181 podman[36774]: 2024-09-21 05:38:19.223307975 +0000 UTC m=+0.339208586 container start 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:38:19.317 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:19 smithi181 ceph-mgr[36793]: -- 172.21.15.181:0/2905650992 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4bcfa30e0 con 0x55a4c1d46000 2024-09-21T05:38:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:19.543 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:19 smithi181 bash[36774]: 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 2024-09-21T05:38:19.544 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:38:19 smithi181 systemd[1]: Started Ceph mgr.z for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:38:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 -- 172.21.15.28:0/743875687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d410de20 msgr2=0x7fd6d4110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/743875687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d410de20 0x7fd6d4110310 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009900 tx=0x7fd6cc02a410 comp rx=0 tx=0).stop 2024-09-21T05:38:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 -- 172.21.15.28:0/743875687 shutdown_connections 2024-09-21T05:38:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/743875687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d410de20 0x7fd6d4110310 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/743875687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d4105310 0x7fd6d410d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/743875687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 0x7fd6d4104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 -- 172.21.15.28:0/743875687 >> 172.21.15.28:0/743875687 conn(0x7fd6d4079710 msgr2=0x7fd6d4079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 -- 172.21.15.28:0/743875687 shutdown_connections 2024-09-21T05:38:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.620+0000 7fd6d9b33640 1 -- 172.21.15.28:0/743875687 wait complete. 2024-09-21T05:38:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 Processor -- start 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 -- start start 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 0x7fd6d419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d410de20 0x7fd6d419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4113840 con 0x7fd6d4105310 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41136c0 con 0x7fd6d41049c0 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.621+0000 7fd6d9b33640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41139c0 con 0x7fd6d410de20 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49524/0 (socket says 172.21.15.28:49524) 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 -- 172.21.15.28:0/1272670683 learned_addr learned my addr 172.21.15.28:0/1272670683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d37fe640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 0x7fd6d419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d3fff640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d410de20 0x7fd6d419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 -- 172.21.15.28:0/1272670683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d410de20 msgr2=0x7fd6d419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d410de20 0x7fd6d419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 -- 172.21.15.28:0/1272670683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 msgr2=0x7fd6d419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 0x7fd6d419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 -- 172.21.15.28:0/1272670683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41a01a0 con 0x7fd6d4105310 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d2ffd640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800d950 tx=0x7fd6c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8017070 con 0x7fd6d4105310 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d41a0490 con 0x7fd6d4105310 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.622+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7fd6c8004490 con 0x7fd6d4105310 2024-09-21T05:38:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.623+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8005550 con 0x7fd6d4105310 2024-09-21T05:38:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.623+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d41a6bf0 con 0x7fd6d4105310 2024-09-21T05:38:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.624+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d406c280 con 0x7fd6d4105310 2024-09-21T05:38:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.624+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7fd6c80047a0 con 0x7fd6d4105310 2024-09-21T05:38:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.624+0000 7fd6d0ff9640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 0x7fd6bc042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.624+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd6c8054250 con 0x7fd6d4105310 2024-09-21T05:38:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.624+0000 7fd6d37fe640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 0x7fd6bc042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:19.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.625+0000 7fd6d37fe640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 0x7fd6bc042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4005fd0 tx=0x7fd6c4009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:19.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.629+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd6c8020020 con 0x7fd6d4105310 2024-09-21T05:38:19.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.819+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7fd6d4108f20 con 0x7fd6d4105310 2024-09-21T05:38:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.823+0000 7fd6d0ff9640 1 -- 172.21.15.28:0/1272670683 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v13)=0 v13) ==== 116+0+0 (secure 0 0 0) 0x7fd6c800e350 con 0x7fd6d4105310 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 msgr2=0x7fd6bc042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 0x7fd6bc042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4005fd0 tx=0x7fd6c4009500 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 msgr2=0x7fd6d419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800d950 tx=0x7fd6c800de10 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 shutdown_connections 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6d410de20 0x7fd6d419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6bc03fb70 0x7fd6bc042080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6d4105310 0x7fd6d419a7d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 --2- 172.21.15.28:0/1272670683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d41049c0 0x7fd6d419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 >> 172.21.15.28:0/1272670683 conn(0x7fd6d4079710 msgr2=0x7fd6d4100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 shutdown_connections 2024-09-21T05:38:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:19.825+0000 7fd6d9b33640 1 -- 172.21.15.28:0/1272670683 wait complete. 2024-09-21T05:38:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 -- 172.21.15.28:0/259531437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c102a60 msgr2=0x7f7c1c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 --2- 172.21.15.28:0/259531437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c102a60 0x7f7c1c102e70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009900 tx=0x7f7c1802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 -- 172.21.15.28:0/259531437 shutdown_connections 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 --2- 172.21.15.28:0/259531437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c108cf0 0x7f7c1c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 --2- 172.21.15.28:0/259531437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 --2- 172.21.15.28:0/259531437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c102a60 0x7f7c1c102e70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 -- 172.21.15.28:0/259531437 >> 172.21.15.28:0/259531437 conn(0x7f7c1c079730 msgr2=0x7f7c1c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 -- 172.21.15.28:0/259531437 shutdown_connections 2024-09-21T05:38:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.349+0000 7f7c23587640 1 -- 172.21.15.28:0/259531437 wait complete. 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 Processor -- start 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 -- start start 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c102a60 0x7f7c1c19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c108cf0 0x7f7c1c19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c112720 con 0x7f7c1c1083a0 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1125a0 con 0x7f7c1c102a60 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.350+0000 7f7c23587640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1128a0 con 0x7f7c1c108cf0 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 -- 172.21.15.28:0/544084571 learned_addr learned my addr 172.21.15.28:0/544084571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c21b24640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c108cf0 0x7f7c1c19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c21323640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c102a60 0x7f7c1c19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 -- 172.21.15.28:0/544084571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c108cf0 msgr2=0x7f7c1c19bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c108cf0 0x7f7c1c19bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 -- 172.21.15.28:0/544084571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c102a60 msgr2=0x7f7c1c19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c102a60 0x7f7c1c19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 -- 172.21.15.28:0/544084571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c1a6360 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c20b22640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00d770 tx=0x7f7c0c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c017070 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f7c0c009c30 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00f3c0 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.351+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c1a6650 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.352+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1a6b40 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7f7c0c01a070 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c102e70 con 0x7f7c1c1083a0 2024-09-21T05:38:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c127fc640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 0x7f7c080420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7c0c054090 con 0x7f7c1c1083a0 2024-09-21T05:38:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c21323640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 0x7f7c080420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.353+0000 7f7c21323640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 0x7f7c080420b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c18000c00 tx=0x7f7c18031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.358+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7c0c01adb0 con 0x7f7c1c1083a0 2024-09-21T05:38:20.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: Deploying daemon mgr.y on smithi129 2024-09-21T05:38:20.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:20 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1272670683' entity='client.admin' 2024-09-21T05:38:20.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.548+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f7c1c10aac0 con 0x7f7c1c1083a0 2024-09-21T05:38:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.552+0000 7f7c127fc640 1 -- 172.21.15.28:0/544084571 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v14)=0 v14) ==== 117+0+0 (secure 0 0 0) 0x7f7c0c015110 con 0x7f7c1c1083a0 2024-09-21T05:38:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 msgr2=0x7f7c080420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 0x7f7c080420b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c18000c00 tx=0x7f7c18031400 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 msgr2=0x7f7c1c19b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00d770 tx=0x7f7c0c00dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 shutdown_connections 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c1c108cf0 0x7f7c1c19bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7c0803fba0 0x7f7c080420b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1083a0 0x7f7c1c19b610 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 --2- 172.21.15.28:0/544084571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c1c102a60 0x7f7c1c19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.554+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 >> 172.21.15.28:0/544084571 conn(0x7f7c1c079730 msgr2=0x7f7c1c1015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.555+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 shutdown_connections 2024-09-21T05:38:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:20.555+0000 7f7c23587640 1 -- 172.21.15.28:0/544084571 wait complete. 2024-09-21T05:38:20.638 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.638 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.638 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: Deploying daemon mgr.y on smithi129 2024-09-21T05:38:20.639 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:20 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1272670683' entity='client.admin' 2024-09-21T05:38:20.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: Deploying daemon mgr.y on smithi129 2024-09-21T05:38:20.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:20 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1272670683' entity='client.admin' 2024-09-21T05:38:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/2592273410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d94102a70 msgr2=0x7f5d94102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/2592273410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d94102a70 0x7f5d94102e80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009900 tx=0x7f5d8802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/2592273410 shutdown_connections 2024-09-21T05:38:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/2592273410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94108d00 0x7f5d9410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/2592273410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d941083b0 0x7f5d941087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/2592273410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d94102a70 0x7f5d94102e80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/2592273410 >> 172.21.15.28:0/2592273410 conn(0x7f5d94079710 msgr2=0x7f5d94079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/2592273410 shutdown_connections 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.055+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/2592273410 wait complete. 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.056+0000 7f5d9b1a4640 1 Processor -- start 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.056+0000 7f5d9b1a4640 1 -- start start 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94102a70 0x7f5d9419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d94108d00 0x7f5d9419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d94112780 con 0x7f5d941083b0 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d94112600 con 0x7f5d94102a70 2024-09-21T05:38:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d94112900 con 0x7f5d94108d00 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49572/0 (socket says 172.21.15.28:49572) 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 -- 172.21.15.28:0/1818033107 learned_addr learned my addr 172.21.15.28:0/1818033107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d98f40640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94102a70 0x7f5d9419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d99741640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d94108d00 0x7f5d9419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 -- 172.21.15.28:0/1818033107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d94108d00 msgr2=0x7f5d9419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d94108d00 0x7f5d9419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 -- 172.21.15.28:0/1818033107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94102a70 msgr2=0x7f5d9419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94102a70 0x7f5d9419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 -- 172.21.15.28:0/1818033107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d941a0290 con 0x7f5d941083b0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d93fff640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000d750 tx=0x7f5d8000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d80017070 con 0x7f5d941083b0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d941a0580 con 0x7f5d941083b0 2024-09-21T05:38:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f5d80005490 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.057+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d80004b10 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.058+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d941a6ce0 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.058+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7f5d80004cb0 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.059+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d94103cf0 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.059+0000 7f5d91ffb640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 0x7f5d84042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.059+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f5d80054100 con 0x7f5d941083b0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.059+0000 7f5d98f40640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 0x7f5d84042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.059+0000 7f5d98f40640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 0x7f5d84042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5d88031440 tx=0x7f5d88009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.064+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5d80009c30 con 0x7f5d941083b0 2024-09-21T05:38:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.257+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f5d9410ba90 con 0x7f5d941083b0 2024-09-21T05:38:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.261+0000 7f5d91ffb640 1 -- 172.21.15.28:0/1818033107 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v15)=0 v15) ==== 108+0+0 (secure 0 0 0) 0x7f5d80022d10 con 0x7f5d941083b0 2024-09-21T05:38:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 msgr2=0x7f5d84042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 0x7f5d84042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5d88031440 tx=0x7f5d88009560 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 msgr2=0x7f5d9419a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000d750 tx=0x7f5d8000dc10 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 shutdown_connections 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d94108d00 0x7f5d9419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d8403fb70 0x7f5d84042080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d941083b0 0x7f5d9419a8c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.263+0000 7f5d9b1a4640 1 --2- 172.21.15.28:0/1818033107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d94102a70 0x7f5d9419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.264+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 >> 172.21.15.28:0/1818033107 conn(0x7f5d94079710 msgr2=0x7f5d94101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.264+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 shutdown_connections 2024-09-21T05:38:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.264+0000 7f5d9b1a4640 1 -- 172.21.15.28:0/1818033107 wait complete. 2024-09-21T05:38:21.283 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 systemd[1]: Starting Ceph mgr.y for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:38:21.547 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 podman[36411]: 2024-09-21 05:38:21.18466871 +0000 UTC m=+0.015561839 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:38:21.547 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 podman[36411]: 2024-09-21 05:38:21.290048819 +0000 UTC m=+0.120941948 container create 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T05:38:21.547 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 podman[36411]: 2024-09-21 05:38:21.509076208 +0000 UTC m=+0.339969337 container init 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:38:21.547 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 podman[36411]: 2024-09-21 05:38:21.511706197 +0000 UTC m=+0.342599329 container start 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-09-21T05:38:21.548 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 ceph-mgr[36430]: -- 172.21.15.129:0/1168681910 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56513e19b0e0 con 0x565142f3e480 2024-09-21T05:38:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 -- 172.21.15.28:0/3415003269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83441083a0 msgr2=0x7f83441087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 --2- 172.21.15.28:0/3415003269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83441083a0 0x7f83441087b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8338009900 tx=0x7f833802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 -- 172.21.15.28:0/3415003269 shutdown_connections 2024-09-21T05:38:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 --2- 172.21.15.28:0/3415003269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344108cf0 0x7f834410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 --2- 172.21.15.28:0/3415003269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83441083a0 0x7f83441087b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 --2- 172.21.15.28:0/3415003269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8344102a60 0x7f8344102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 -- 172.21.15.28:0/3415003269 >> 172.21.15.28:0/3415003269 conn(0x7f8344079700 msgr2=0x7f8344079b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 -- 172.21.15.28:0/3415003269 shutdown_connections 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.748+0000 7f8349dee640 1 -- 172.21.15.28:0/3415003269 wait complete. 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 Processor -- start 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 -- start start 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83441083a0 0x7f834419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 0x7f834419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8344112770 con 0x7f8344108cf0 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f83441125f0 con 0x7f8344102a60 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.749+0000 7f8349dee640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f83441128f0 con 0x7f83441083a0 2024-09-21T05:38:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 0x7f834419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33442/0 (socket says 172.21.15.28:33442) 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 0x7f834419fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49606/0 (socket says 172.21.15.28:49606) 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 -- 172.21.15.28:0/1747586783 learned_addr learned my addr 172.21.15.28:0/1747586783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8342ffd640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83441083a0 0x7f834419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 -- 172.21.15.28:0/1747586783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83441083a0 msgr2=0x7f834419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83441083a0 0x7f834419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 -- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 msgr2=0x7f834419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 0x7f834419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 -- 172.21.15.28:0/1747586783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83441a0280 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f83437fe640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f833400e9f0 tx=0x7f833400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833400cd80 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f8334005490 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.750+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83441a0570 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.751+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334010430 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.751+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83440ff4d0 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.752+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8344103ce0 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.752+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7f83340105d0 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.752+0000 7f8340ff9640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 0x7f832c042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.752+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8334054a80 con 0x7f8344102a60 2024-09-21T05:38:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.752+0000 7f8342ffd640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 0x7f832c042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.753+0000 7f8342ffd640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 0x7f832c042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8338009230 tx=0x7f833803c040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.758+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8334016330 con 0x7f8344102a60 2024-09-21T05:38:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:21 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/544084571' entity='client.admin' 2024-09-21T05:38:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:21 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1818033107' entity='client.admin' 2024-09-21T05:38:21.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:21 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/544084571' entity='client.admin' 2024-09-21T05:38:21.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:21 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1818033107' entity='client.admin' 2024-09-21T05:38:21.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 bash[36411]: 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 2024-09-21T05:38:21.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:38:21 smithi129 systemd[1]: Started Ceph mgr.y for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:38:21.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.889+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 7 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f833405a080 con 0x7f8344102a60 2024-09-21T05:38:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 -- 172.21.15.28:0/646560027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec10def0 msgr2=0x7f9cec1103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/646560027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec10def0 0x7f9cec1103e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009ca0 tx=0x7f9ce402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 -- 172.21.15.28:0/646560027 shutdown_connections 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/646560027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec10def0 0x7f9cec1103e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/646560027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 0x7f9cec10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/646560027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec104a90 0x7f9cec104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 -- 172.21.15.28:0/646560027 >> 172.21.15.28:0/646560027 conn(0x7f9cec079750 msgr2=0x7f9cec079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.652+0000 7f9cf43ac640 1 -- 172.21.15.28:0/646560027 shutdown_connections 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.653+0000 7f9cf43ac640 1 -- 172.21.15.28:0/646560027 wait complete. 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.653+0000 7f9cf43ac640 1 Processor -- start 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.653+0000 7f9cf43ac640 1 -- start start 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 0x7f9cec19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113820 con 0x7f9cec104a90 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1136a0 con 0x7f9cec1053e0 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf43ac640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1139a0 con 0x7f9cec10def0 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2148640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33954/0 (socket says 172.21.15.28:33954) 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2148640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49584/0 (socket says 172.21.15.28:49584) 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 -- 172.21.15.28:0/2069553964 learned_addr learned my addr 172.21.15.28:0/2069553964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf1947640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 0x7f9cec19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 -- 172.21.15.28:0/2069553964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 msgr2=0x7f9cec19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 0x7f9cec19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 -- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 msgr2=0x7f9cec19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2949640 1 -- 172.21.15.28:0/2069553964 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec1a0280 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.654+0000 7f9cf2148640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9cf2949640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4005cc0 tx=0x7f9ce4009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4003bf0 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9ce4003d90 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4030a00 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec1a0510 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.655+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1a6cd0 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.656+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec06c240 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.656+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7f9ce4030bc0 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.656+0000 7f9ce37fe640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 0x7f9cd8042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.657+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f9ce40759e0 con 0x7f9cec10def0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.657+0000 7f9cf2148640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 0x7f9cd8042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.657+0000 7f9cf2148640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 0x7f9cd8042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9cec06c940 tx=0x7f9ce8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.662+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9ce402edb0 con 0x7f9cec10def0 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.884+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f9cec108ff0 con 0x7f9cec10def0 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.889+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 7 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ce40360c0 con 0x7f9cec10def0 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.889+0000 7f9ce37fe640 1 -- 172.21.15.28:0/2069553964 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7f9ce403b410 con 0x7f9cec10def0 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.891+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 msgr2=0x7f9cd8042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.891+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 0x7f9cd8042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9cec06c940 tx=0x7f9ce8005f90 comp rx=0 tx=0).stop 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.891+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 msgr2=0x7f9cec19fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.891+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4005cc0 tx=0x7f9ce4009880 comp rx=0 tx=0).stop 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 shutdown_connections 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cec10def0 0x7f9cec19fb50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cec1053e0 0x7f9cec19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9cd803fb70 0x7f9cd8042080 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 --2- 172.21.15.28:0/2069553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec104a90 0x7f9cec19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 >> 172.21.15.28:0/2069553964 conn(0x7f9cec079750 msgr2=0x7f9cec0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 shutdown_connections 2024-09-21T05:38:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.892+0000 7f9cf43ac640 1 -- 172.21.15.28:0/2069553964 wait complete. 2024-09-21T05:38:21.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.951+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f834410baf0 con 0x7f8344102a60 2024-09-21T05:38:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.959+0000 7f8340ff9640 1 -- 172.21.15.28:0/1747586783 <== mon.2 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v17)=0 v17) ==== 106+0+0 (secure 0 0 0) 0x7f83340108c0 con 0x7f8344102a60 2024-09-21T05:38:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 msgr2=0x7f832c042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 0x7f832c042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8338009230 tx=0x7f833803c040 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 msgr2=0x7f834419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f833400e9f0 tx=0x7f833400edf0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 shutdown_connections 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f832c03fb70 0x7f832c042080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8344108cf0 0x7f834419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83441083a0 0x7f834419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 --2- 172.21.15.28:0/1747586783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8344102a60 0x7f834419a310 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.961+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 >> 172.21.15.28:0/1747586783 conn(0x7f8344079700 msgr2=0x7f8344101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.962+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 shutdown_connections 2024-09-21T05:38:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:21.962+0000 7f8349dee640 1 -- 172.21.15.28:0/1747586783 wait complete. 2024-09-21T05:38:22.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:21 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/544084571' entity='client.admin' 2024-09-21T05:38:22.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:21 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1818033107' entity='client.admin' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:22 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:22 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:23.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:22 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:23.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: Reconfiguring daemon mgr.x on smithi028 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:23.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4252703808' entity='client.admin' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:24.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: Reconfiguring daemon mgr.x on smithi028 2024-09-21T05:38:24.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:23 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4252703808' entity='client.admin' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: Reconfiguring daemon mgr.x on smithi028 2024-09-21T05:38:24.071 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:24.072 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:23 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4252703808' entity='client.admin' 2024-09-21T05:38:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.390+0000 7fcceb4ad640 1 -- 172.21.15.28:0/2329544232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4102650 msgr2=0x7fcce4102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.390+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/2329544232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4102650 0x7fcce4102a60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009900 tx=0x7fcce002a480 comp rx=0 tx=0).stop 2024-09-21T05:38:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 -- 172.21.15.28:0/2329544232 shutdown_connections 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/2329544232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4110200 0x7fcce4112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/2329544232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 0x7fcce410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/2329544232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4102650 0x7fcce4102a60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 -- 172.21.15.28:0/2329544232 >> 172.21.15.28:0/2329544232 conn(0x7fcce4079760 msgr2=0x7fcce4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 -- 172.21.15.28:0/2329544232 shutdown_connections 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.391+0000 7fcceb4ad640 1 -- 172.21.15.28:0/2329544232 wait complete. 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 Processor -- start 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 -- start start 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4102650 0x7fcce419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 0x7fcce419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcce4115c70 con 0x7fcce4110200 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcce4115af0 con 0x7fcce4102fa0 2024-09-21T05:38:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcceb4ad640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcce4115df0 con 0x7fcce4102650 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49652/0 (socket says 172.21.15.28:49652) 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9249640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4102650 0x7fcce419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 -- 172.21.15.28:0/3647939026 learned_addr learned my addr 172.21.15.28:0/3647939026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce8a48640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 0x7fcce419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 -- 172.21.15.28:0/3647939026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4102650 msgr2=0x7fcce419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4102650 0x7fcce419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 -- 172.21.15.28:0/3647939026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 msgr2=0x7fcce419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.392+0000 7fcce9a4a640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 0x7fcce419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fcce9a4a640 1 -- 172.21.15.28:0/3647939026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce41a25b0 con 0x7fcce4110200 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fcce9a4a640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00eb10 tx=0x7fccdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc004b20 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccdc004cc0 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce41a28a0 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc010670 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.393+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce41a9000 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.394+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7fccdc010830 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.394+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce406c1f0 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.394+0000 7fccda7fc640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 0x7fccd0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.394+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fccdc054c90 con 0x7fcce4110200 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.394+0000 7fcce9249640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 0x7fccd0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.395+0000 7fcce9249640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 0x7fccd0042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcce0031440 tx=0x7fcce0009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.399+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fccdc016330 con 0x7fcce4110200 2024-09-21T05:38:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.588+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fcce41a9380 con 0x7fcce4110200 2024-09-21T05:38:24.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.591+0000 7fccda7fc640 1 -- 172.21.15.28:0/3647939026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v19)=0 v19) ==== 111+0+0 (secure 0 0 0) 0x7fccdc019d50 con 0x7fcce4110200 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 msgr2=0x7fccd0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 0x7fccd0042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcce0031440 tx=0x7fcce0009560 comp rx=0 tx=0).stop 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 msgr2=0x7fcce41a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00eb10 tx=0x7fccdc00c2d0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 shutdown_connections 2024-09-21T05:38:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd003fb70 0x7fccd0042080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4110200 0x7fcce41a1e80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce4102fa0 0x7fcce419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 --2- 172.21.15.28:0/3647939026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce4102650 0x7fcce419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 >> 172.21.15.28:0/3647939026 conn(0x7fcce4079760 msgr2=0x7fcce4109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 shutdown_connections 2024-09-21T05:38:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:24.594+0000 7fcceb4ad640 1 -- 172.21.15.28:0/3647939026 wait complete. 2024-09-21T05:38:24.821 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: Standby manager daemon z started 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:25.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:24 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3647939026' entity='client.admin' 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: Standby manager daemon z started 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:25.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:24 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3647939026' entity='client.admin' 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: Standby manager daemon z started 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.181:0/993954185' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:25.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:24 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3647939026' entity='client.admin' 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.305+0000 7f62f420f640 1 -- 172.21.15.28:0/3895167832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 msgr2=0x7f62ec103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.305+0000 7f62f420f640 1 --2- 172.21.15.28:0/3895167832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec103f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009900 tx=0x7f62e802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 -- 172.21.15.28:0/3895167832 shutdown_connections 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 --2- 172.21.15.28:0/3895167832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec109df0 0x7f62ec110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 --2- 172.21.15.28:0/3895167832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec1094a0 0x7f62ec1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 --2- 172.21.15.28:0/3895167832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec103f70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 -- 172.21.15.28:0/3895167832 >> 172.21.15.28:0/3895167832 conn(0x7f62ec07b850 msgr2=0x7f62ec07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 -- 172.21.15.28:0/3895167832 shutdown_connections 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.306+0000 7f62f420f640 1 -- 172.21.15.28:0/3895167832 wait complete. 2024-09-21T05:38:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 Processor -- start 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 -- start start 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec1094a0 0x7f62ec19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec109df0 0x7f62ec19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec113990 con 0x7f62ec103b60 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec113810 con 0x7f62ec1094a0 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f420f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec113b10 con 0x7f62ec109df0 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49630/0 (socket says 172.21.15.28:49630) 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 -- 172.21.15.28:0/4252703808 learned_addr learned my addr 172.21.15.28:0/4252703808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f27ac640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec109df0 0x7f62ec19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f17aa640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec1094a0 0x7f62ec19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 -- 172.21.15.28:0/4252703808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec109df0 msgr2=0x7f62ec19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec109df0 0x7f62ec19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 -- 172.21.15.28:0/4252703808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec1094a0 msgr2=0x7f62ec19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec1094a0 0x7f62ec19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.307+0000 7f62f1fab640 1 -- 172.21.15.28:0/4252703808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec1a0370 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62f1fab640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62e8006010 tx=0x7f62e80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e803f030 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f62e80044c0 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec1a0600 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8035070 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.308+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1a6dc0 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.309+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 51915+0+0 (secure 0 0 0) 0x7f62e8042030 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.309+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec103f70 con 0x7f62ec103b60 2024-09-21T05:38:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.309+0000 7f62e2ffd640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 0x7f62d8042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.309+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f62e80749d0 con 0x7f62ec103b60 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.309+0000 7f62f17aa640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 0x7f62d8042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.310+0000 7f62f17aa640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 0x7f62d8042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f62ec106660 tx=0x7f62dc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.314+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f62e80433c0 con 0x7f62ec103b60 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.504+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f62ec02e640 con 0x7f62ec103b60 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.507+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v18)=0 v18) ==== 123+0+0 (secure 0 0 0) 0x7f62e8003ca0 con 0x7f62ec103b60 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.508+0000 7f62e2ffd640 1 -- 172.21.15.28:0/4252703808 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62e8043740 con 0x7f62ec103b60 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 msgr2=0x7f62d8042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 0x7f62d8042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f62ec106660 tx=0x7f62dc005f90 comp rx=0 tx=0).stop 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 msgr2=0x7f62ec19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62e8006010 tx=0x7f62e80040c0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 shutdown_connections 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62ec109df0 0x7f62ec19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec1094a0 0x7f62ec19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d803fb70 0x7f62d8042080 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 --2- 172.21.15.28:0/4252703808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62ec103b60 0x7f62ec19a400 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 >> 172.21.15.28:0/4252703808 conn(0x7f62ec07b850 msgr2=0x7f62ec107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 shutdown_connections 2024-09-21T05:38:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:23.510+0000 7f62f420f640 1 -- 172.21.15.28:0/4252703808 wait complete. 2024-09-21T05:38:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:25 smithi028 ceph-mon[31227]: mgrmap e19: x(active, since 12s), standbys: z 2024-09-21T05:38:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T05:38:25.966 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:25.966 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:25.966 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi028') 2024-09-21T05:38:25.966 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi028') 2024-09-21T05:38:26.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:25 smithi181 ceph-mon[34819]: mgrmap e19: x(active, since 12s), standbys: z 2024-09-21T05:38:26.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T05:38:26.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:25 smithi129 ceph-mon[34885]: mgrmap e19: x(active, since 12s), standbys: z 2024-09-21T05:38:26.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T05:38:26.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:26 smithi028 ceph-mon[31227]: Standby manager daemon y started 2024-09-21T05:38:26.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:26 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T05:38:26.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:26 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:26.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:26 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T05:38:26.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:26 smithi028 ceph-mon[31227]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/2493731727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48108d10 msgr2=0x7fdf4810f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/2493731727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48108d10 0x7fdf4810f540 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009ca0 tx=0x7fdf4002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/2493731727 shutdown_connections 2024-09-21T05:38:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/2493731727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48108d10 0x7fdf4810f540 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/2493731727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 0x7fdf481087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/2493731727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48102a80 0x7fdf48102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/2493731727 >> 172.21.15.28:0/2493731727 conn(0x7fdf48079740 msgr2=0x7fdf48079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/2493731727 shutdown_connections 2024-09-21T05:38:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/2493731727 wait complete. 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 Processor -- start 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.943+0000 7fdf4ec7d640 1 -- start start 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 0x7fdf4819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48108d10 0x7fdf4819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48112980 con 0x7fdf48102a80 2024-09-21T05:38:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48112800 con 0x7fdf481083c0 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ec7d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48112b00 con 0x7fdf48108d10 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40468/0 (socket says 172.21.15.28:40468) 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 -- 172.21.15.28:0/3960527518 learned_addr learned my addr 172.21.15.28:0/3960527518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 0x7fdf4819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4d21a640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48108d10 0x7fdf4819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 -- 172.21.15.28:0/3960527518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48108d10 msgr2=0x7fdf4819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48108d10 0x7fdf4819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 -- 172.21.15.28:0/3960527518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 msgr2=0x7fdf4819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 0x7fdf4819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 -- 172.21.15.28:0/3960527518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf481a03b0 con 0x7fdf48102a80 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf4ca19640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d770 tx=0x7fdf3800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.944+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf38017070 con 0x7fdf48102a80 2024-09-21T05:38:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.945+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf481a06a0 con 0x7fdf48102a80 2024-09-21T05:38:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.945+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf380054d0 con 0x7fdf48102a80 2024-09-21T05:38:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.945+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf38004c30 con 0x7fdf48102a80 2024-09-21T05:38:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.945+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481a6e00 con 0x7fdf48102a80 2024-09-21T05:38:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.946+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf48103d00 con 0x7fdf48102a80 2024-09-21T05:38:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.947+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fdf38004dd0 con 0x7fdf48102a80 2024-09-21T05:38:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.948+0000 7fdf45ffb640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 0x7fdf300b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.948+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 0x7fdf300b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.948+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fdf380e55d0 con 0x7fdf48102a80 2024-09-21T05:38:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.949+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 0x7fdf300b89e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf48105ea0 tx=0x7fdf3c00b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:26.951+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdf380ada60 con 0x7fdf48102a80 2024-09-21T05:38:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:27.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:26 smithi181 ceph-mon[34819]: Standby manager daemon y started 2024-09-21T05:38:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:26 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T05:38:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:26 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:26 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T05:38:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:26 smithi181 ceph-mon[34819]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:27.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:26 smithi129 ceph-mon[34885]: Standby manager daemon y started 2024-09-21T05:38:27.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:26 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T05:38:27.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:26 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T05:38:27.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:26 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T05:38:27.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:26 smithi129 ceph-mon[34885]: from='mgr.? 172.21.15.129:0/1894597817' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T05:38:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fdf4819b570 con 0x7fdf48102a80 2024-09-21T05:38:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/2236615057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 msgr2=0x7fc7d41087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/2236615057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d41087c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009900 tx=0x7fc7c402a410 comp rx=0 tx=0).stop 2024-09-21T05:38:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/2236615057 shutdown_connections 2024-09-21T05:38:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/2236615057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4108d00 0x7fc7d410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/2236615057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d41087c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/2236615057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4102a70 0x7fc7d4102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/2236615057 >> 172.21.15.28:0/2236615057 conn(0x7fc7d4079790 msgr2=0x7fc7d4079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/2236615057 shutdown_connections 2024-09-21T05:38:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.151+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/2236615057 wait complete. 2024-09-21T05:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.152+0000 7fc7dbbc0640 1 Processor -- start 2024-09-21T05:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.152+0000 7fc7dbbc0640 1 -- start start 2024-09-21T05:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.152+0000 7fc7dbbc0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7dbbc0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d40ffb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7dbbc0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4108d00 0x7fc7d4100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7dbbc0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41129c0 con 0x7fc7d41083b0 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7dbbc0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112840 con 0x7fc7d4102a70 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7dbbc0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112b40 con 0x7fc7d4108d00 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:44224/0 (socket says 172.21.15.28:44224) 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 -- 172.21.15.28:0/1317717596 learned_addr learned my addr 172.21.15.28:0/1317717596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7da15d640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4108d00 0x7fc7d4100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d915b640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d40ffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 -- 172.21.15.28:0/1317717596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4108d00 msgr2=0x7fc7d4100070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4108d00 0x7fc7d4100070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 -- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 msgr2=0x7fc7d40ffb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d40ffb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 -- 172.21.15.28:0/1317717596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d41a8780 con 0x7fc7d4102a70 2024-09-21T05:38:27.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.153+0000 7fc7d995c640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000d770 tx=0x7fc7d000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:27.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.155+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3960527518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v20)=0 v20) ==== 125+0+0 (secure 0 0 0) 0x7fdf380b1e40 con 0x7fdf48102a80 2024-09-21T05:38:27.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.156+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0017070 con 0x7fc7d4102a70 2024-09-21T05:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.156+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d00054d0 con 0x7fc7d4102a70 2024-09-21T05:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.156+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d41a8a70 con 0x7fc7d4102a70 2024-09-21T05:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.156+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0004c30 con 0x7fc7d4102a70 2024-09-21T05:38:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.156+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a8f80 con 0x7fc7d4102a70 2024-09-21T05:38:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.157+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d4103cf0 con 0x7fc7d4102a70 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.157+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 msgr2=0x7fdf300b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.157+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 0x7fdf300b89e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf48105ea0 tx=0x7fdf3c00b410 comp rx=0 tx=0).stop 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.157+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 msgr2=0x7fdf4819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.157+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d770 tx=0x7fdf3800dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 shutdown_connections 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf48108d10 0x7fdf4819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf481083c0 0x7fdf4819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf300b64d0 0x7fdf300b89e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 --2- 172.21.15.28:0/3960527518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf48102a80 0x7fdf4819a440 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 >> 172.21.15.28:0/3960527518 conn(0x7fdf48079740 msgr2=0x7fdf481019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 shutdown_connections 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.158+0000 7fdf4ec7d640 1 -- 172.21.15.28:0/3960527518 wait complete. 2024-09-21T05:38:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.159+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fc7d0004e10 con 0x7fc7d4102a70 2024-09-21T05:38:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.160+0000 7fc7caffd640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 0x7fc7c00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.160+0000 7fc7d915b640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 0x7fc7c00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.160+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc7d00e5100 con 0x7fc7d4102a70 2024-09-21T05:38:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.160+0000 7fc7d915b640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 0x7fc7c00b8a60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009230 tx=0x7fc7c4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:27.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.164+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc7d00b1970 con 0x7fc7d4102a70 2024-09-21T05:38:27.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.355+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7fc7d410bae0 con 0x7fc7d4102a70 2024-09-21T05:38:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.361+0000 7fc7caffd640 1 -- 172.21.15.28:0/1317717596 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v21)=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7fc7d00b68b0 con 0x7fc7d4102a70 2024-09-21T05:38:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.363+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 msgr2=0x7fc7c00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.363+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 0x7fc7c00b8a60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009230 tx=0x7fc7c4031400 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.363+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 msgr2=0x7fc7d40ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.363+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000d770 tx=0x7fc7d000dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 shutdown_connections 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d4108d00 0x7fc7d4100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7c00b6550 0x7fc7c00b8a60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d41083b0 0x7fc7d40ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 --2- 172.21.15.28:0/1317717596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d4102a70 0x7fc7d40ff5f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 >> 172.21.15.28:0/1317717596 conn(0x7fc7d4079790 msgr2=0x7fc7d406bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 shutdown_connections 2024-09-21T05:38:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:27.364+0000 7fc7dbbc0640 1 -- 172.21.15.28:0/1317717596 wait complete. 2024-09-21T05:38:27.585 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:27.586 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:27.587 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:27.587 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:27.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:27 smithi028 ceph-mon[31227]: mgrmap e20: x(active, since 14s), standbys: z, y 2024-09-21T05:38:27.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T05:38:27.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3960527518' entity='client.admin' 2024-09-21T05:38:27.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:27 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:27 smithi181 ceph-mon[34819]: mgrmap e20: x(active, since 14s), standbys: z, y 2024-09-21T05:38:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T05:38:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3960527518' entity='client.admin' 2024-09-21T05:38:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:27 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:27 smithi129 ceph-mon[34885]: mgrmap e20: x(active, since 14s), standbys: z, y 2024-09-21T05:38:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T05:38:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3960527518' entity='client.admin' 2024-09-21T05:38:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:27 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 -- 172.21.15.28:0/1718231237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8111cb0 msgr2=0x7f62e81120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 --2- 172.21.15.28:0/1718231237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8111cb0 0x7f62e81120c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009900 tx=0x7f62d002a410 comp rx=0 tx=0).stop 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 -- 172.21.15.28:0/1718231237 shutdown_connections 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 --2- 172.21.15.28:0/1718231237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 0x7f62e8118e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 --2- 172.21.15.28:0/1718231237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8111cb0 0x7f62e81120c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 --2- 172.21.15.28:0/1718231237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e810c370 0x7f62e810c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.140+0000 7f62eccba640 1 -- 172.21.15.28:0/1718231237 >> 172.21.15.28:0/1718231237 conn(0x7f62e8071a80 msgr2=0x7f62e810a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.141+0000 7f62eccba640 1 -- 172.21.15.28:0/1718231237 shutdown_connections 2024-09-21T05:38:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.141+0000 7f62eccba640 1 -- 172.21.15.28:0/1718231237 wait complete. 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.141+0000 7f62eccba640 1 Processor -- start 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.141+0000 7f62eccba640 1 -- start start 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 0x7f62e81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 0x7f62e81a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62e811c2c0 con 0x7f62e810c370 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62e811c140 con 0x7f62e8112600 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62eccba640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f62e811c440 con 0x7f62e8111cb0 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40514/0 (socket says 172.21.15.28:40514) 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 -- 172.21.15.28:0/2805149925 learned_addr learned my addr 172.21.15.28:0/2805149925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e6d9d640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 0x7f62e81a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e5d9b640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 0x7f62e81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 -- 172.21.15.28:0/2805149925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 msgr2=0x7f62e81a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 0x7f62e81a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 -- 172.21.15.28:0/2805149925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 msgr2=0x7f62e81a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 0x7f62e81a85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 -- 172.21.15.28:0/2805149925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e81a8c20 con 0x7f62e810c370 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e5d9b640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 0x7f62e81a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.142+0000 7f62e659c640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c920 tx=0x7f62d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.143+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8007c70 con 0x7f62e810c370 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.143+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62d8007e10 con 0x7f62e810c370 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.143+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e81a8f10 con 0x7f62e810c370 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.143+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8018820 con 0x7f62e810c370 2024-09-21T05:38:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.143+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e81af730 con 0x7f62e810c370 2024-09-21T05:38:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.144+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e810c780 con 0x7f62e810c370 2024-09-21T05:38:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.146+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f62d800f450 con 0x7f62e810c370 2024-09-21T05:38:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.147+0000 7f62df7fe640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 0x7f62d40b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.147+0000 7f62e5d9b640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 0x7f62d40b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.147+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f62d80e5a70 con 0x7f62e810c370 2024-09-21T05:38:28.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.147+0000 7f62e5d9b640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 0x7f62d40b8a90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62d002aa00 tx=0x7f62d0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.150+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f62d8014030 con 0x7f62e810c370 2024-09-21T05:38:28.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.343+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f62e81a3ea0 con 0x7f62e810c370 2024-09-21T05:38:28.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.347+0000 7f62df7fe640 1 -- 172.21.15.28:0/2805149925 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v22)=0 v22) ==== 127+0+0 (secure 0 0 0) 0x7f62d80ae010 con 0x7f62e810c370 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 msgr2=0x7f62d40b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 0x7f62d40b8a90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62d002aa00 tx=0x7f62d0031400 comp rx=0 tx=0).stop 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 msgr2=0x7f62e81a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c920 tx=0x7f62d800cde0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 shutdown_connections 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62e8112600 0x7f62e81a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62e8111cb0 0x7f62e81a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f62d40b6580 0x7f62d40b8a90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 --2- 172.21.15.28:0/2805149925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e810c370 0x7f62e81a2d70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 >> 172.21.15.28:0/2805149925 conn(0x7f62e8071a80 msgr2=0x7f62e80722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 shutdown_connections 2024-09-21T05:38:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.350+0000 7f62eccba640 1 -- 172.21.15.28:0/2805149925 wait complete. 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 -- 172.21.15.28:0/4090203461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 msgr2=0x7f8bd81098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/4090203461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd81098c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009900 tx=0x7f8bc402a410 comp rx=0 tx=0).stop 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 -- 172.21.15.28:0/4090203461 shutdown_connections 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/4090203461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/4090203461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd81098c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/4090203461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 0x7f8bd8103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 -- 172.21.15.28:0/4090203461 >> 172.21.15.28:0/4090203461 conn(0x7f8bd807b910 msgr2=0x7f8bd807bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 -- 172.21.15.28:0/4090203461 shutdown_connections 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.379+0000 7f8bdeb50640 1 -- 172.21.15.28:0/4090203461 wait complete. 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 Processor -- start 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 -- start start 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 0x7f8bd8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113ac0 con 0x7f8bd81094b0 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113940 con 0x7f8bd8109e00 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.380+0000 7f8bdeb50640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113c40 con 0x7f8bd8103bc0 2024-09-21T05:38:28.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40532/0 (socket says 172.21.15.28:40532) 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 -- 172.21.15.28:0/1509482943 learned_addr learned my addr 172.21.15.28:0/1509482943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bdc8ec640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 0x7f8bd8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bdd0ed640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 -- 172.21.15.28:0/1509482943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 msgr2=0x7f8bd8075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 0x7f8bd8075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 -- 172.21.15.28:0/1509482943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 msgr2=0x7f8bd8076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 -- 172.21.15.28:0/1509482943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd81a8750 con 0x7f8bd81094b0 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bdd0ed640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd7fff640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bc402a930 tx=0x7f8bc4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc403f030 con 0x7f8bd81094b0 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd81a89e0 con 0x7f8bd81094b0 2024-09-21T05:38:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc4003aa0 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.381+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4035070 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.382+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a8ec0 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.383+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd8103fd0 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.384+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f8bc4042030 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.385+0000 7f8bd5ffb640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 0x7f8bc80b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.385+0000 7f8bdc8ec640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 0x7f8bc80b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.385+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8bc4105e40 con 0x7f8bd81094b0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.386+0000 7f8bdc8ec640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 0x7f8bc80b89e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8077160 tx=0x7f8bcc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.389+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8bc40ce2d0 con 0x7f8bd81094b0 2024-09-21T05:38:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.429 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.436 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:28.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.604+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f8bd802e640 con 0x7f8bd81094b0 2024-09-21T05:38:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.608+0000 7f8bd5ffb640 1 -- 172.21.15.28:0/1509482943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v23)=0 v23) ==== 116+0+0 (secure 0 0 0) 0x7f8bc40d26b0 con 0x7f8bd81094b0 2024-09-21T05:38:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 msgr2=0x7f8bc80b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 0x7f8bc80b89e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8077160 tx=0x7f8bcc00b390 comp rx=0 tx=0).stop 2024-09-21T05:38:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 msgr2=0x7f8bd8075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bc402a930 tx=0x7f8bc4004270 comp rx=0 tx=0).stop 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 shutdown_connections 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd8109e00 0x7f8bd8076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bc80b64d0 0x7f8bc80b89e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd81094b0 0x7f8bd8075c50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 --2- 172.21.15.28:0/1509482943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd8103bc0 0x7f8bd8075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 >> 172.21.15.28:0/1509482943 conn(0x7f8bd807b910 msgr2=0x7f8bd81075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 shutdown_connections 2024-09-21T05:38:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:28.610+0000 7f8bdeb50640 1 -- 172.21.15.28:0/1509482943 wait complete. 2024-09-21T05:38:29.279 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi028') 2024-09-21T05:38:29.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2805149925' entity='client.admin' 2024-09-21T05:38:29.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1509482943' entity='client.admin' 2024-09-21T05:38:29.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2805149925' entity='client.admin' 2024-09-21T05:38:29.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1509482943' entity='client.admin' 2024-09-21T05:38:29.799 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2805149925' entity='client.admin' 2024-09-21T05:38:29.799 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1509482943' entity='client.admin' 2024-09-21T05:38:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 -- 172.21.15.28:0/1885365467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 msgr2=0x7fe29410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 --2- 172.21.15.28:0/1885365467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29410f550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009ca0 tx=0x7fe28c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 -- 172.21.15.28:0/1885365467 shutdown_connections 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 --2- 172.21.15.28:0/1885365467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29410f550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 --2- 172.21.15.28:0/1885365467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 0x7fe2941087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 --2- 172.21.15.28:0/1885365467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 0x7fe294102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 -- 172.21.15.28:0/1885365467 >> 172.21.15.28:0/1885365467 conn(0x7fe294079760 msgr2=0x7fe294079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 -- 172.21.15.28:0/1885365467 shutdown_connections 2024-09-21T05:38:29.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.933+0000 7fe29be29640 1 -- 172.21.15.28:0/1885365467 wait complete. 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.934+0000 7fe29be29640 1 Processor -- start 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.934+0000 7fe29be29640 1 -- start start 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 0x7fe2941994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 0x7fe294199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe294112990 con 0x7fe294108d20 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe294112810 con 0x7fe2941083d0 2024-09-21T05:38:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29be29640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe294112b10 con 0x7fe294102ae0 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40558/0 (socket says 172.21.15.28:40558) 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 -- 172.21.15.28:0/3514407752 learned_addr learned my addr 172.21.15.28:0/3514407752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe299bc5640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 0x7fe2941994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe2993c4640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 0x7fe294199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 -- 172.21.15.28:0/3514407752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 msgr2=0x7fe2941994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 0x7fe2941994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 -- 172.21.15.28:0/3514407752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 msgr2=0x7fe294199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 0x7fe294199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:29.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 -- 172.21.15.28:0/3514407752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29419f410 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.935+0000 7fe29a3c6640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005cc0 tx=0x7fe28c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.936+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003bf0 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.936+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28c003d90 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.936+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29419f6a0 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.936+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c02fbc0 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.936+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2941a5e60 con 0x7fe294108d20 2024-09-21T05:38:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.937+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe294103d10 con 0x7fe294108d20 2024-09-21T05:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.938+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fe28c02fd60 con 0x7fe294108d20 2024-09-21T05:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.939+0000 7fe286ffd640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 0x7fe2800b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.939+0000 7fe299bc5640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 0x7fe2800b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.939+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe28c106c00 con 0x7fe294108d20 2024-09-21T05:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.940+0000 7fe299bc5640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 0x7fe2800b89e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe290005d00 tx=0x7fe29000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:29.943+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe28c0cf090 con 0x7fe294108d20 2024-09-21T05:38:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.132+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7fe29410ab10 con 0x7fe294108d20 2024-09-21T05:38:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.136+0000 7fe286ffd640 1 -- 172.21.15.28:0/3514407752 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v24)=0 v24) ==== 122+0+0 (secure 0 0 0) 0x7fe28c0098b0 con 0x7fe294108d20 2024-09-21T05:38:30.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 msgr2=0x7fe2800b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 0x7fe2800b89e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe290005d00 tx=0x7fe29000b410 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 msgr2=0x7fe29419ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005cc0 tx=0x7fe28c02f380 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 shutdown_connections 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe2800b64d0 0x7fe2800b89e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe294108d20 0x7fe29419ece0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2941083d0 0x7fe294199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 --2- 172.21.15.28:0/3514407752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe294102ae0 0x7fe2941994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.138+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 >> 172.21.15.28:0/3514407752 conn(0x7fe294079760 msgr2=0x7fe2941044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.139+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 shutdown_connections 2024-09-21T05:38:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:30.139+0000 7fe29be29640 1 -- 172.21.15.28:0/3514407752 wait complete. 2024-09-21T05:38:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 -- 172.21.15.28:0/2054895316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff690109df0 msgr2=0x7ff690110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2054895316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff690109df0 0x7ff690110620 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff688009900 tx=0x7ff68802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 -- 172.21.15.28:0/2054895316 shutdown_connections 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2054895316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff690109df0 0x7ff690110620 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2054895316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901094a0 0x7ff6901098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2054895316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 0x7ff690103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 -- 172.21.15.28:0/2054895316 >> 172.21.15.28:0/2054895316 conn(0x7ff69007b850 msgr2=0x7ff69007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 -- 172.21.15.28:0/2054895316 shutdown_connections 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 -- 172.21.15.28:0/2054895316 wait complete. 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.130+0000 7ff6953bc640 1 Processor -- start 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 -- start start 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 0x7ff69019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 0x7ff69019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff690113ab0 con 0x7ff6901094a0 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff690113930 con 0x7ff690103b60 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff6953bc640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff690113c30 con 0x7ff690109df0 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40584/0 (socket says 172.21.15.28:40584) 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 -- 172.21.15.28:0/2034001669 learned_addr learned my addr 172.21.15.28:0/2034001669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 0x7ff69019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68f7fe640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 0x7ff69019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 -- 172.21.15.28:0/2034001669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 msgr2=0x7ff69019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 0x7ff69019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 -- 172.21.15.28:0/2034001669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 msgr2=0x7ff69019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 0x7ff69019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 -- 172.21.15.28:0/2034001669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6901a02b0 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68f7fe640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 0x7ff69019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.131+0000 7ff68e7fc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff68400d950 tx=0x7ff68400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.132+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684017070 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.132+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6901a05a0 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.132+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff684009c30 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.132+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6840047a0 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.132+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6901a6dc0 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.133+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff690103f70 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.134+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7ff684004020 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.135+0000 7ff677fff640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 0x7ff6780b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.135+0000 7ff68effd640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 0x7ff6780b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.135+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff6840e57c0 con 0x7ff6901094a0 2024-09-21T05:38:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.136+0000 7ff68effd640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 0x7ff6780b8a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff680007ec0 tx=0x7ff6800075b0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.138+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff684020020 con 0x7ff6901094a0 2024-09-21T05:38:31.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:31 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3514407752' entity='client.admin' 2024-09-21T05:38:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.328+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7ff69019b530 con 0x7ff6901094a0 2024-09-21T05:38:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.328+0000 7ff677fff640 1 -- 172.21.15.28:0/2034001669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v24)=0 v24) ==== 108+0+0 (secure 0 0 0) 0x7ff6840adc50 con 0x7ff6901094a0 2024-09-21T05:38:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 msgr2=0x7ff6780b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 0x7ff6780b8a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff680007ec0 tx=0x7ff6800075b0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 msgr2=0x7ff69019a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff68400d950 tx=0x7ff68400de10 comp rx=0 tx=0).stop 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 shutdown_connections 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff690109df0 0x7ff69019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6780b6500 0x7ff6780b8a10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6901094a0 0x7ff69019a9a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 --2- 172.21.15.28:0/2034001669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff690103b60 0x7ff69019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.330+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 >> 172.21.15.28:0/2034001669 conn(0x7ff69007b850 msgr2=0x7ff690107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.331+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 shutdown_connections 2024-09-21T05:38:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:31.331+0000 7ff6953bc640 1 -- 172.21.15.28:0/2034001669 wait complete. 2024-09-21T05:38:31.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:31 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3514407752' entity='client.admin' 2024-09-21T05:38:31.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:31 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3514407752' entity='client.admin' 2024-09-21T05:38:32.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 -- 172.21.15.28:0/1482160323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a100760a0 msgr2=0x7f3a1010fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/1482160323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a100760a0 0x7f3a1010fc00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3a04009ca0 tx=0x7f3a0402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 -- 172.21.15.28:0/1482160323 shutdown_connections 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/1482160323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 0x7f3a10112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/1482160323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a100760a0 0x7f3a1010fc00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/1482160323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a10075750 0x7f3a10075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 -- 172.21.15.28:0/1482160323 >> 172.21.15.28:0/1482160323 conn(0x7f3a100ff590 msgr2=0x7f3a101019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 -- 172.21.15.28:0/1482160323 shutdown_connections 2024-09-21T05:38:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.988+0000 7f3a14dc2640 1 -- 172.21.15.28:0/1482160323 wait complete. 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.989+0000 7f3a14dc2640 1 Processor -- start 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.989+0000 7f3a14dc2640 1 -- start start 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.989+0000 7f3a14dc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.989+0000 7f3a14dc2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a100760a0 0x7f3a1019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a14dc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 0x7f3a101a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a14dc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a10115cb0 con 0x7f3a10075750 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a14dc2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a10115b30 con 0x7f3a10110140 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a14dc2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a10115e30 con 0x7f3a100760a0 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40600/0 (socket says 172.21.15.28:40600) 2024-09-21T05:38:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 -- 172.21.15.28:0/2396317440 learned_addr learned my addr 172.21.15.28:0/2396317440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0dd9b640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a100760a0 0x7f3a1019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0ed9d640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 0x7f3a101a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 -- 172.21.15.28:0/2396317440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a100760a0 msgr2=0x7f3a1019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a100760a0 0x7f3a1019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 -- 172.21.15.28:0/2396317440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 msgr2=0x7f3a101a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 0x7f3a101a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 -- 172.21.15.28:0/2396317440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a101a2650 con 0x7f3a10075750 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f3a0e59c640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000e9f0 tx=0x7f3a0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.990+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000cd80 con 0x7f3a10075750 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.991+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a000054d0 con 0x7f3a10075750 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.991+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00010690 con 0x7f3a10075750 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.991+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a101a2940 con 0x7f3a10075750 2024-09-21T05:38:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.991+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a101a90a0 con 0x7f3a10075750 2024-09-21T05:38:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.992+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a10075b60 con 0x7f3a10075750 2024-09-21T05:38:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.993+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f3a00004e30 con 0x7f3a10075750 2024-09-21T05:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.994+0000 7f39ff7fe640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 0x7f39f80b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.994+0000 7f3a0dd9b640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 0x7f39f80b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.994+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f3a000e5c60 con 0x7f3a10075750 2024-09-21T05:38:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.995+0000 7f3a0dd9b640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 0x7f39f80b8a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a04000f80 tx=0x7f3a04009360 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:32.997+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3a000ae0f0 con 0x7f3a10075750 2024-09-21T05:38:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.186+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f3a101a9380 con 0x7f3a10075750 2024-09-21T05:38:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.190+0000 7f39ff7fe640 1 -- 172.21.15.28:0/2396317440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v25)=0 v25) ==== 120+0+0 (secure 0 0 0) 0x7f3a000b24d0 con 0x7f3a10075750 2024-09-21T05:38:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 msgr2=0x7f39f80b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 0x7f39f80b8a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a04000f80 tx=0x7f3a04009360 comp rx=0 tx=0).stop 2024-09-21T05:38:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 msgr2=0x7f3a1019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000e9f0 tx=0x7f3a0000edf0 comp rx=0 tx=0).stop 2024-09-21T05:38:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 shutdown_connections 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a10110140 0x7f3a101a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a100760a0 0x7f3a1019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f39f80b6550 0x7f39f80b8a60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 --2- 172.21.15.28:0/2396317440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a10075750 0x7f3a1019c6e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 >> 172.21.15.28:0/2396317440 conn(0x7f3a100ff590 msgr2=0x7f3a10109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 shutdown_connections 2024-09-21T05:38:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.193+0000 7f3a14dc2640 1 -- 172.21.15.28:0/2396317440 wait complete. 2024-09-21T05:38:33.907 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.999+0000 7f5abc460640 1 -- 172.21.15.28:0/1599270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4103b90 msgr2=0x7f5ab4103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.999+0000 7f5abc460640 1 --2- 172.21.15.28:0/1599270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4103b90 0x7f5ab4103fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009900 tx=0x7f5ab002a410 comp rx=0 tx=0).stop 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.999+0000 7f5abc460640 1 -- 172.21.15.28:0/1599270836 shutdown_connections 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.999+0000 7f5abc460640 1 --2- 172.21.15.28:0/1599270836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab4109e20 0x7f5ab4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.000+0000 7f5abc460640 1 --2- 172.21.15.28:0/1599270836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab41094d0 0x7f5ab41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.000+0000 7f5abc460640 1 --2- 172.21.15.28:0/1599270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4103b90 0x7f5ab4103fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.000+0000 7f5abc460640 1 -- 172.21.15.28:0/1599270836 >> 172.21.15.28:0/1599270836 conn(0x7f5ab40ff600 msgr2=0x7f5ab4101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.000+0000 7f5abc460640 1 -- 172.21.15.28:0/1599270836 shutdown_connections 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.000+0000 7f5abc460640 1 -- 172.21.15.28:0/1599270836 wait complete. 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 Processor -- start 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 -- start start 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab4103b90 0x7f5ab419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 0x7f5ab419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4113ae0 con 0x7f5ab4109e20 2024-09-21T05:38:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4113960 con 0x7f5ab4103b90 2024-09-21T05:38:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5abc460640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4113c60 con 0x7f5ab41094d0 2024-09-21T05:38:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5aba9fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5aba9fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40644/0 (socket says 172.21.15.28:40644) 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5aba9fd640 1 -- 172.21.15.28:0/2246448679 learned_addr learned my addr 172.21.15.28:0/2246448679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5aba1fc640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab4103b90 0x7f5ab419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.001+0000 7f5ab99fb640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 0x7f5ab419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 -- 172.21.15.28:0/2246448679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 msgr2=0x7f5ab419aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 0x7f5ab419aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 -- 172.21.15.28:0/2246448679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab4103b90 msgr2=0x7f5ab419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab4103b90 0x7f5ab419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 -- 172.21.15.28:0/2246448679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab41a0440 con 0x7f5ab4109e20 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5ab99fb640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 0x7f5ab419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aba9fd640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eaa0 tx=0x7f5aac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00cdb0 con 0x7f5ab4109e20 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aac005020 con 0x7f5ab4109e20 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab41a0730 con 0x7f5ab4109e20 2024-09-21T05:38:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac0116d0 con 0x7f5ab4109e20 2024-09-21T05:38:34.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.002+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a6e90 con 0x7f5ab4109e20 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.004+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab4104e10 con 0x7f5ab4109e20 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.005+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f5aac0051d0 con 0x7f5ab4109e20 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.006+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 0x7f5aa00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.006+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f5aac0e58d0 con 0x7f5ab4109e20 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.006+0000 7f5aba1fc640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 0x7f5aa00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.007+0000 7f5aba1fc640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 0x7f5aa00b8a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0031440 tx=0x7f5ab0009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:34.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.011+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5aac016030 con 0x7f5ab4109e20 2024-09-21T05:38:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 -- 172.21.15.28:0/3663819421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70102aa0 msgr2=0x7fbf70102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 --2- 172.21.15.28:0/3663819421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70102aa0 0x7fbf70102eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009ca0 tx=0x7fbf6c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 -- 172.21.15.28:0/3663819421 shutdown_connections 2024-09-21T05:38:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 --2- 172.21.15.28:0/3663819421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf70108d30 0x7fbf7010f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 --2- 172.21.15.28:0/3663819421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf701083e0 0x7fbf701087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 --2- 172.21.15.28:0/3663819421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70102aa0 0x7fbf70102eb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 -- 172.21.15.28:0/3663819421 >> 172.21.15.28:0/3663819421 conn(0x7fbf70079750 msgr2=0x7fbf70079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 -- 172.21.15.28:0/3663819421 shutdown_connections 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.923+0000 7fbf78088640 1 -- 172.21.15.28:0/3663819421 wait complete. 2024-09-21T05:38:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 Processor -- start 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 -- start start 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 0x7fbf70199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 0x7fbf701999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf701129f0 con 0x7fbf70108d30 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf70112870 con 0x7fbf701083e0 2024-09-21T05:38:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.924+0000 7fbf78088640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbf70112b70 con 0x7fbf70102aa0 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf76625640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf76625640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40628/0 (socket says 172.21.15.28:40628) 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf76625640 1 -- 172.21.15.28:0/1626647025 learned_addr learned my addr 172.21.15.28:0/1626647025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75623640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 0x7fbf701999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 0x7fbf70199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 -- 172.21.15.28:0/1626647025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 msgr2=0x7fbf701999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 0x7fbf701999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 -- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 msgr2=0x7fbf7019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75e24640 1 -- 172.21.15.28:0/1626647025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7019f370 con 0x7fbf70102aa0 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf75623640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 0x7fbf701999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.925+0000 7fbf76625640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf75e24640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 0x7fbf70199400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c02ade0 tx=0x7fbf6c031660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003b70 con 0x7fbf70102aa0 2024-09-21T05:38:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf6c031910 con 0x7fbf70102aa0 2024-09-21T05:38:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7019f600 con 0x7fbf70102aa0 2024-09-21T05:38:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02f950 con 0x7fbf70102aa0 2024-09-21T05:38:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.926+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7006b820 con 0x7fbf70102aa0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.928+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf70102eb0 con 0x7fbf70102aa0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.929+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fbf6c004040 con 0x7fbf70102aa0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.930+0000 7fbf66ffd640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 0x7fbf5c0b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.930+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbf6c1066c0 con 0x7fbf70102aa0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.930+0000 7fbf75623640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 0x7fbf5c0b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.930+0000 7fbf75623640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 0x7fbf5c0b8ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbf70104ab0 tx=0x7fbf60012040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:33.934+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbf6c0ceab0 con 0x7fbf70102aa0 2024-09-21T05:38:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.138+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7fbf7006c1b0 con 0x7fbf70102aa0 2024-09-21T05:38:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.144+0000 7fbf66ffd640 1 -- 172.21.15.28:0/1626647025 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v26)=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7fbf6c0d2e90 con 0x7fbf70102aa0 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.146+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 msgr2=0x7fbf5c0b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.146+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 0x7fbf5c0b8ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbf70104ab0 tx=0x7fbf60012040 comp rx=0 tx=0).stop 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 msgr2=0x7fbf70199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 0x7fbf70199400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c02ade0 tx=0x7fbf6c031660 comp rx=0 tx=0).stop 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 shutdown_connections 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf5c0b65d0 0x7fbf5c0b8ae0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf70108d30 0x7fbf7019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf701083e0 0x7fbf701999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 --2- 172.21.15.28:0/1626647025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf70102aa0 0x7fbf70199400 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 >> 172.21.15.28:0/1626647025 conn(0x7fbf70079750 msgr2=0x7fbf701064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:34.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 shutdown_connections 2024-09-21T05:38:34.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.147+0000 7fbf78088640 1 -- 172.21.15.28:0/1626647025 wait complete. 2024-09-21T05:38:34.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.226+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f5ab402e640 con 0x7f5ab4109e20 2024-09-21T05:38:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.230+0000 7f5aab7fe640 1 -- 172.21.15.28:0/2246448679 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v27)=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7f5aac0adcc0 con 0x7f5ab4109e20 2024-09-21T05:38:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.232+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 msgr2=0x7f5aa00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.232+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 0x7f5aa00b8a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0031440 tx=0x7f5ab0009560 comp rx=0 tx=0).stop 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.232+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 msgr2=0x7f5ab419fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.232+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eaa0 tx=0x7f5aac00ef60 comp rx=0 tx=0).stop 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 shutdown_connections 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5aa00b6550 0x7f5aa00b8a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab4109e20 0x7f5ab419fd10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab41094d0 0x7f5ab419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 --2- 172.21.15.28:0/2246448679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab4103b90 0x7f5ab419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 >> 172.21.15.28:0/2246448679 conn(0x7f5ab40ff600 msgr2=0x7f5ab4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 shutdown_connections 2024-09-21T05:38:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:34.233+0000 7f5abc460640 1 -- 172.21.15.28:0/2246448679 wait complete. 2024-09-21T05:38:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:34 smithi028 ceph-mon[31227]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:34 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2396317440' entity='client.admin' 2024-09-21T05:38:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:34 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' 2024-09-21T05:38:34.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:34 smithi181 ceph-mon[34819]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:34 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2396317440' entity='client.admin' 2024-09-21T05:38:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:34 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' 2024-09-21T05:38:34.554 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:34 smithi129 ceph-mon[34885]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:34.554 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:34 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2396317440' entity='client.admin' 2024-09-21T05:38:34.554 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:34 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' 2024-09-21T05:38:35.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:35 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2246448679' entity='client.admin' 2024-09-21T05:38:35.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:35 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2246448679' entity='client.admin' 2024-09-21T05:38:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:35 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2246448679' entity='client.admin' 2024-09-21T05:38:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 -- 172.21.15.28:0/1752597240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f41094a0 msgr2=0x7ff6f41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/1752597240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f41094a0 0x7ff6f41098b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009900 tx=0x7ff6e802a3f0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 -- 172.21.15.28:0/1752597240 shutdown_connections 2024-09-21T05:38:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/1752597240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f4109df0 0x7ff6f4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/1752597240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f41094a0 0x7ff6f41098b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/1752597240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4103b60 0x7ff6f4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 -- 172.21.15.28:0/1752597240 >> 172.21.15.28:0/1752597240 conn(0x7ff6f407b850 msgr2=0x7ff6f407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 -- 172.21.15.28:0/1752597240 shutdown_connections 2024-09-21T05:38:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.929+0000 7ff6f88fd640 1 -- 172.21.15.28:0/1752597240 wait complete. 2024-09-21T05:38:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 Processor -- start 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 -- start start 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f41094a0 0x7ff6f419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4109df0 0x7ff6f419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f4113ab0 con 0x7ff6f4103b60 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f4113930 con 0x7ff6f4109df0 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.930+0000 7ff6f88fd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f4113c30 con 0x7ff6f41094a0 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39604/0 (socket says 172.21.15.28:39604) 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 -- 172.21.15.28:0/3271305110 learned_addr learned my addr 172.21.15.28:0/3271305110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f2d9d640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4109df0 0x7ff6f419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f1d9b640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f41094a0 0x7ff6f419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 -- 172.21.15.28:0/3271305110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f41094a0 msgr2=0x7ff6f419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f41094a0 0x7ff6f419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 -- 172.21.15.28:0/3271305110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4109df0 msgr2=0x7ff6f419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4109df0 0x7ff6f419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 -- 172.21.15.28:0/3271305110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f41a0370 con 0x7ff6f4103b60 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f259c640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c920 tx=0x7ff6e400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4007c70 con 0x7ff6f4103b60 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e4007e10 con 0x7ff6f4103b60 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4018820 con 0x7ff6f4103b60 2024-09-21T05:38:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f41a0660 con 0x7ff6f4103b60 2024-09-21T05:38:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.931+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f41a6e80 con 0x7ff6f4103b60 2024-09-21T05:38:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.932+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f4104de0 con 0x7ff6f4103b60 2024-09-21T05:38:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.934+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7ff6e400f450 con 0x7ff6f4103b60 2024-09-21T05:38:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.935+0000 7ff6e37fe640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 0x7ff6dc0b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.935+0000 7ff6f1d9b640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 0x7ff6dc0b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.935+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff6e40e5b90 con 0x7ff6f4103b60 2024-09-21T05:38:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.935+0000 7ff6f1d9b640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 0x7ff6dc0b8a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009230 tx=0x7ff6e8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:35.938+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6e4014030 con 0x7ff6f4103b60 2024-09-21T05:38:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.128+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7ff6f4103f70 con 0x7ff6f4103b60 2024-09-21T05:38:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.132+0000 7ff6e37fe640 1 -- 172.21.15.28:0/3271305110 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v28)=0 v28) ==== 121+0+0 (secure 0 0 0) 0x7ff6e40ae0b0 con 0x7ff6f4103b60 2024-09-21T05:38:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 msgr2=0x7ff6dc0b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 0x7ff6dc0b8a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009230 tx=0x7ff6e8031400 comp rx=0 tx=0).stop 2024-09-21T05:38:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 msgr2=0x7ff6f419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c920 tx=0x7ff6e400cde0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 shutdown_connections 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f4109df0 0x7ff6f419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f41094a0 0x7ff6f419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6dc0b6500 0x7ff6dc0b8a10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 --2- 172.21.15.28:0/3271305110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6f4103b60 0x7ff6f419a4c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 >> 172.21.15.28:0/3271305110 conn(0x7ff6f407b850 msgr2=0x7ff6f4105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.134+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 shutdown_connections 2024-09-21T05:38:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.135+0000 7ff6f88fd640 1 -- 172.21.15.28:0/3271305110 wait complete. 2024-09-21T05:38:36.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 -- 172.21.15.28:0/2823711460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1160104840 msgr2=0x7f1160104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 --2- 172.21.15.28:0/2823711460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1160104840 0x7f1160104c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1158009900 tx=0x7f115802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 -- 172.21.15.28:0/2823711460 shutdown_connections 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 --2- 172.21.15.28:0/2823711460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f116010ddf0 0x7f1160110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 --2- 172.21.15.28:0/2823711460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116010d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 --2- 172.21.15.28:0/2823711460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1160104840 0x7f1160104c50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.207+0000 7f11670c8640 1 -- 172.21.15.28:0/2823711460 >> 172.21.15.28:0/2823711460 conn(0x7f1160079730 msgr2=0x7f1160079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.208+0000 7f11670c8640 1 -- 172.21.15.28:0/2823711460 shutdown_connections 2024-09-21T05:38:36.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.208+0000 7f11670c8640 1 -- 172.21.15.28:0/2823711460 wait complete. 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.208+0000 7f11670c8640 1 Processor -- start 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.208+0000 7f11670c8640 1 -- start start 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1160104840 0x7f116019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113a90 con 0x7f116010ddf0 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113910 con 0x7f1160105270 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f11670c8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113c10 con 0x7f1160104840 2024-09-21T05:38:36.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39620/0 (socket says 172.21.15.28:39620) 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 -- 172.21.15.28:0/2544827934 learned_addr learned my addr 172.21.15.28:0/2544827934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f115ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1164e64640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1160104840 0x7f116019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 -- 172.21.15.28:0/2544827934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1160104840 msgr2=0x7f116019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1160104840 0x7f116019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 -- 172.21.15.28:0/2544827934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 msgr2=0x7f116019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 -- 172.21.15.28:0/2544827934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11601a0430 con 0x7f116010ddf0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f115ffff640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.209+0000 7f1165665640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f115000eb10 tx=0x7f115000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.210+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115000ce20 con 0x7f116010ddf0 2024-09-21T05:38:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.210+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1150005020 con 0x7f116010ddf0 2024-09-21T05:38:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.210+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11601a0720 con 0x7f116010ddf0 2024-09-21T05:38:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.210+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150010690 con 0x7f116010ddf0 2024-09-21T05:38:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.210+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11601a6f40 con 0x7f116010ddf0 2024-09-21T05:38:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.211+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1160104c50 con 0x7f116010ddf0 2024-09-21T05:38:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.213+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f1150005510 con 0x7f116010ddf0 2024-09-21T05:38:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.213+0000 7f115dffb640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 0x7f114c0b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.213+0000 7f1164e64640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 0x7f114c0b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.214+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f11500e60f0 con 0x7f116010ddf0 2024-09-21T05:38:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.214+0000 7f1164e64640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 0x7f114c0b8a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1158000c00 tx=0x7f1158031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:36.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.216+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f11500ae580 con 0x7f116010ddf0 2024-09-21T05:38:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.408+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f116019b6b0 con 0x7f116010ddf0 2024-09-21T05:38:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.412+0000 7f115dffb640 1 -- 172.21.15.28:0/2544827934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v29)=0 v29) ==== 121+0+0 (secure 0 0 0) 0x7f116019b6b0 con 0x7f116010ddf0 2024-09-21T05:38:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 msgr2=0x7f114c0b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 0x7f114c0b8a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1158000c00 tx=0x7f1158031400 comp rx=0 tx=0).stop 2024-09-21T05:38:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 msgr2=0x7f116019fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f115000eb10 tx=0x7f115000ef10 comp rx=0 tx=0).stop 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 shutdown_connections 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f114c0b6500 0x7f114c0b8a10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116010ddf0 0x7f116019fdc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1160105270 0x7f116019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 --2- 172.21.15.28:0/2544827934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1160104840 0x7f116019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 >> 172.21.15.28:0/2544827934 conn(0x7f1160079730 msgr2=0x7f1160107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 shutdown_connections 2024-09-21T05:38:36.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:36.414+0000 7f11670c8640 1 -- 172.21.15.28:0/2544827934 wait complete. 2024-09-21T05:38:36.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:36 smithi028 ceph-mon[31227]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:36.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3271305110' entity='client.admin' 2024-09-21T05:38:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:36.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:36 smithi181 ceph-mon[34819]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:36.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:36 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3271305110' entity='client.admin' 2024-09-21T05:38:36.565 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi028') 2024-09-21T05:38:36.565 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi028') 2024-09-21T05:38:36.565 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi028') 2024-09-21T05:38:36.565 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi028') 2024-09-21T05:38:36.565 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-21T05:38:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:36 smithi129 ceph-mon[34885]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:36 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3271305110' entity='client.admin' 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 -- 172.21.15.28:0/4094974142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 msgr2=0x7f88641087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4094974142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f88641087c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8854009900 tx=0x7f885402a410 comp rx=0 tx=0).stop 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 -- 172.21.15.28:0/4094974142 shutdown_connections 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4094974142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4094974142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f88641087c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4094974142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 0x7f8864102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 -- 172.21.15.28:0/4094974142 >> 172.21.15.28:0/4094974142 conn(0x7f8864079710 msgr2=0x7f8864079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 -- 172.21.15.28:0/4094974142 shutdown_connections 2024-09-21T05:38:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.537+0000 7f886b3d8640 1 -- 172.21.15.28:0/4094974142 wait complete. 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 Processor -- start 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 -- start start 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 0x7f886419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886419bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88641129c0 con 0x7f88641083b0 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8864112840 con 0x7f8864108d00 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.538+0000 7f886b3d8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8864112b40 con 0x7f8864102a70 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39636/0 (socket says 172.21.15.28:39636) 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8869975640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886419bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 -- 172.21.15.28:0/4035069987 learned_addr learned my addr 172.21.15.28:0/4035069987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8869174640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 0x7f886419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 -- 172.21.15.28:0/4035069987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 msgr2=0x7f886419b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 0x7f886419b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 -- 172.21.15.28:0/4035069987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 msgr2=0x7f886419bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886419bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 -- 172.21.15.28:0/4035069987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88641a6560 con 0x7f88641083b0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8869975640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886419bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f8868973640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f885402a930 tx=0x7f8854004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885403f030 con 0x7f88641083b0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8854003aa0 con 0x7f88641083b0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88641a67f0 con 0x7f88641083b0 2024-09-21T05:38:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854035070 con 0x7f88641083b0 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.539+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88641a6d20 con 0x7f88641083b0 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.541+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8864103cf0 con 0x7f88641083b0 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.542+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f8854042030 con 0x7f88641083b0 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.543+0000 7f885a7fc640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 0x7f88500b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.543+0000 7f8869174640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 0x7f88500b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.543+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8854105d60 con 0x7f88641083b0 2024-09-21T05:38:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.544+0000 7f8869174640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 0x7f88500b8a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88600058c0 tx=0x7f886000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.548+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f88540d25d0 con 0x7f88641083b0 2024-09-21T05:38:37.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:37 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2544827934' entity='client.admin' 2024-09-21T05:38:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.738+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f886402e640 con 0x7f88641083b0 2024-09-21T05:38:37.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.742+0000 7f885a7fc640 1 -- 172.21.15.28:0/4035069987 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v30)=0 v30) ==== 121+0+0 (secure 0 0 0) 0x7f88540d7510 con 0x7f88641083b0 2024-09-21T05:38:37.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.744+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 msgr2=0x7f88500b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.744+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 0x7f88500b8a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88600058c0 tx=0x7f886000a2d0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.744+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 msgr2=0x7f886419b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.744+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f885402a930 tx=0x7f8854004270 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 shutdown_connections 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8864108d00 0x7f886419bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f88500b6580 0x7f88500b8a90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88641083b0 0x7f886419b810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 --2- 172.21.15.28:0/4035069987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8864102a70 0x7f886419b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 >> 172.21.15.28:0/4035069987 conn(0x7f8864079710 msgr2=0x7f88641043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 shutdown_connections 2024-09-21T05:38:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.745+0000 7f886b3d8640 1 -- 172.21.15.28:0/4035069987 wait complete. 2024-09-21T05:38:37.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:37 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2544827934' entity='client.admin' 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- 172.21.15.28:0/4165852951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458102ae0 msgr2=0x7fb458102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/4165852951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458102ae0 0x7fb458102ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb448009900 tx=0x7fb44802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- 172.21.15.28:0/4165852951 shutdown_connections 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/4165852951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458108d20 0x7fb45810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/4165852951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4581083d0 0x7fb4581087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/4165852951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458102ae0 0x7fb458102ef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- 172.21.15.28:0/4165852951 >> 172.21.15.28:0/4165852951 conn(0x7fb458079760 msgr2=0x7fb458079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- 172.21.15.28:0/4165852951 shutdown_connections 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- 172.21.15.28:0/4165852951 wait complete. 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 Processor -- start 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.815+0000 7fb45d6d2640 1 -- start start 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 0x7fb458199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 0x7fb45819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb458112990 con 0x7fb4581083d0 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb458112810 con 0x7fb458108d20 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb45d6d2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb458112b10 con 0x7fb458102ae0 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39652/0 (socket says 172.21.15.28:39652) 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 -- 172.21.15.28:0/3533312732 learned_addr learned my addr 172.21.15.28:0/3533312732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb4577fe640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 0x7fb45819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb456ffd640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 0x7fb458199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 -- 172.21.15.28:0/3533312732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 msgr2=0x7fb458199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 0x7fb458199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 -- 172.21.15.28:0/3533312732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 msgr2=0x7fb45819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 0x7fb45819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 -- 172.21.15.28:0/3533312732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45819f2d0 con 0x7fb4581083d0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb4577fe640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 0x7fb45819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb456ffd640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 0x7fb458199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.816+0000 7fb44ffff640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d770 tx=0x7fb44000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.817+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440017070 con 0x7fb4581083d0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.817+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45819f5c0 con 0x7fb4581083d0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.817+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb440009c30 con 0x7fb4581083d0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.817+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44000f620 con 0x7fb4581083d0 2024-09-21T05:38:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.817+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4581a5d20 con 0x7fb4581083d0 2024-09-21T05:38:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.818+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb458103d10 con 0x7fb4581083d0 2024-09-21T05:38:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.820+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fb44001a070 con 0x7fb4581083d0 2024-09-21T05:38:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.820+0000 7fb454ff9640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 0x7fb4440b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.820+0000 7fb456ffd640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 0x7fb4440b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.820+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb4400e52b0 con 0x7fb4581083d0 2024-09-21T05:38:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.821+0000 7fb456ffd640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 0x7fb4440b89e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb448031440 tx=0x7fb448009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:37.823+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb440009de0 con 0x7fb4581083d0 2024-09-21T05:38:37.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:37 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2544827934' entity='client.admin' 2024-09-21T05:38:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.015+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fb45810aaf0 con 0x7fb4581083d0 2024-09-21T05:38:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.019+0000 7fb454ff9640 1 -- 172.21.15.28:0/3533312732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v31)=0 v31) ==== 119+0+0 (secure 0 0 0) 0x7fb4400ad740 con 0x7fb4581083d0 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.021+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 msgr2=0x7fb4440b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.021+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 0x7fb4440b89e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb448031440 tx=0x7fb448009680 comp rx=0 tx=0).stop 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.021+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 msgr2=0x7fb458199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.021+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d770 tx=0x7fb44000dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 shutdown_connections 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d20 0x7fb45819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b64d0 0x7fb4440b89e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083d0 0x7fb458199900 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 --2- 172.21.15.28:0/3533312732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102ae0 0x7fb458199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 >> 172.21.15.28:0/3533312732 conn(0x7fb458079760 msgr2=0x7fb4581014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 shutdown_connections 2024-09-21T05:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.022+0000 7fb45d6d2640 1 -- 172.21.15.28:0/3533312732 wait complete. 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 -- 172.21.15.28:0/331974068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 msgr2=0x7f94501087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 --2- 172.21.15.28:0/331974068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501087b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9440009900 tx=0x7f944002a410 comp rx=0 tx=0).stop 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 -- 172.21.15.28:0/331974068 shutdown_connections 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 --2- 172.21.15.28:0/331974068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450108cf0 0x7f945010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 --2- 172.21.15.28:0/331974068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501087b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 --2- 172.21.15.28:0/331974068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450102a60 0x7f9450102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 -- 172.21.15.28:0/331974068 >> 172.21.15.28:0/331974068 conn(0x7f9450079730 msgr2=0x7f9450079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 -- 172.21.15.28:0/331974068 shutdown_connections 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.005+0000 7f94580ba640 1 -- 172.21.15.28:0/331974068 wait complete. 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.006+0000 7f94580ba640 1 Processor -- start 2024-09-21T05:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.006+0000 7f94580ba640 1 -- start start 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.006+0000 7f94580ba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450102a60 0x7f9450199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f94580ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f94580ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450108cf0 0x7f945019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f94580ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9450112960 con 0x7f94501083a0 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f94580ba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94501127e0 con 0x7f9450102a60 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f94580ba640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9450112ae0 con 0x7f9450108cf0 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39670/0 (socket says 172.21.15.28:39670) 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 -- 172.21.15.28:0/2218482389 learned_addr learned my addr 172.21.15.28:0/2218482389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455e56640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450102a60 0x7f9450199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9456657640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450108cf0 0x7f945019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 -- 172.21.15.28:0/2218482389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450108cf0 msgr2=0x7f945019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450108cf0 0x7f945019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 -- 172.21.15.28:0/2218482389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450102a60 msgr2=0x7f9450199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450102a60 0x7f9450199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 -- 172.21.15.28:0/2218482389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945019f2c0 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9455655640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9440032fd0 tx=0x7f9440004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.007+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944003f030 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.008+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9440004490 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.008+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945019f550 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.008+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440035070 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.008+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f945019fa80 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.009+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9450102e70 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.010+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f9440042030 con 0x7f94501083a0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.011+0000 7f9446ffd640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 0x7f943c0b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.011+0000 7f9455e56640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 0x7f943c0b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.011+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f9440105c20 con 0x7f94501083a0 2024-09-21T05:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.012+0000 7f9455e56640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 0x7f943c0b8b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f944c005fd0 tx=0x7f944c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.015+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f945019fa80 con 0x7f94501083a0 2024-09-21T05:38:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.211+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f945010a590 con 0x7f94501083a0 2024-09-21T05:38:38.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.215+0000 7f9446ffd640 1 -- 172.21.15.28:0/2218482389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v32)=0 v32) ==== 119+0+0 (secure 0 0 0) 0x7f94400ce010 con 0x7f94501083a0 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.217+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 msgr2=0x7f943c0b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.217+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 0x7f943c0b8b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f944c005fd0 tx=0x7f944c00b390 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.217+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 msgr2=0x7f94501999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.217+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9440032fd0 tx=0x7f9440004270 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 shutdown_connections 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9450108cf0 0x7f945019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f943c0b6600 0x7f943c0b8b10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94501083a0 0x7f94501999b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 --2- 172.21.15.28:0/2218482389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9450102a60 0x7f9450199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 >> 172.21.15.28:0/2218482389 conn(0x7f9450079730 msgr2=0x7f94501044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 shutdown_connections 2024-09-21T05:38:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:38.218+0000 7f94580ba640 1 -- 172.21.15.28:0/2218482389 wait complete. 2024-09-21T05:38:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:38 smithi028 ceph-mon[31227]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4035069987' entity='client.admin' 2024-09-21T05:38:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3533312732' entity='client.admin' 2024-09-21T05:38:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2218482389' entity='client.admin' 2024-09-21T05:38:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:38 smithi181 ceph-mon[34819]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4035069987' entity='client.admin' 2024-09-21T05:38:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3533312732' entity='client.admin' 2024-09-21T05:38:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2218482389' entity='client.admin' 2024-09-21T05:38:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:38 smithi129 ceph-mon[34885]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4035069987' entity='client.admin' 2024-09-21T05:38:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3533312732' entity='client.admin' 2024-09-21T05:38:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2218482389' entity='client.admin' 2024-09-21T05:38:40.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:40 smithi028 ceph-mon[31227]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:40 smithi181 ceph-mon[34819]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:40 smithi129 ceph-mon[34885]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 -- 172.21.15.28:0/1364637709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 msgr2=0x7f4ed4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/1364637709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed4102e70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009900 tx=0x7f4ed002a3f0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 -- 172.21.15.28:0/1364637709 shutdown_connections 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/1364637709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed4108cf0 0x7f4ed410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/1364637709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed41083a0 0x7f4ed41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/1364637709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed4102e70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 -- 172.21.15.28:0/1364637709 >> 172.21.15.28:0/1364637709 conn(0x7f4ed4079730 msgr2=0x7f4ed4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 -- 172.21.15.28:0/1364637709 shutdown_connections 2024-09-21T05:38:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.146+0000 7f4edc2bf640 1 -- 172.21.15.28:0/1364637709 wait complete. 2024-09-21T05:38:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.147+0000 7f4edc2bf640 1 Processor -- start 2024-09-21T05:38:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.147+0000 7f4edc2bf640 1 -- start start 2024-09-21T05:38:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed41083a0 0x7f4ed419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed4108cf0 0x7f4ed419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed4112960 con 0x7f4ed4102a60 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41127e0 con 0x7f4ed4108cf0 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4edc2bf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed4112ae0 con 0x7f4ed41083a0 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39698/0 (socket says 172.21.15.28:39698) 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 -- 172.21.15.28:0/499040365 learned_addr learned my addr 172.21.15.28:0/499040365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4ed985a640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed41083a0 0x7f4ed419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda85c640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed4108cf0 0x7f4ed419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 -- 172.21.15.28:0/499040365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed41083a0 msgr2=0x7f4ed419b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed41083a0 0x7f4ed419b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 -- 172.21.15.28:0/499040365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed4108cf0 msgr2=0x7f4ed419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed4108cf0 0x7f4ed419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.148+0000 7f4eda05b640 1 -- 172.21.15.28:0/499040365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed41a6490 con 0x7f4ed4102a60 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4eda05b640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0005940 tx=0x7f4ed00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed003f030 con 0x7f4ed4102a60 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ed0003cb0 con 0x7f4ed4102a60 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed41a6720 con 0x7f4ed4102a60 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0035070 con 0x7f4ed4102a60 2024-09-21T05:38:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.149+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed41a6c00 con 0x7f4ed4102a60 2024-09-21T05:38:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.150+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed4102e70 con 0x7f4ed4102a60 2024-09-21T05:38:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.151+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f4ed0042030 con 0x7f4ed4102a60 2024-09-21T05:38:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.152+0000 7f4ecb7fe640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 0x7f4ec00b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.152+0000 7f4ed985a640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 0x7f4ec00b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.152+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4ed0105980 con 0x7f4ed4102a60 2024-09-21T05:38:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.153+0000 7f4ed985a640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 0x7f4ec00b8a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4100c50 tx=0x7f4ec400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.157+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4ed00cdd70 con 0x7f4ed4102a60 2024-09-21T05:38:41.307 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi028') 2024-09-21T05:38:41.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.347+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f4ed410a530 con 0x7f4ed4102a60 2024-09-21T05:38:41.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.351+0000 7f4ecb7fe640 1 -- 172.21.15.28:0/499040365 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7f4ed00d2150 con 0x7f4ed4102a60 2024-09-21T05:38:41.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 msgr2=0x7f4ec00b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 0x7f4ec00b8a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4100c50 tx=0x7f4ec400b410 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 msgr2=0x7f4ed419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0005940 tx=0x7f4ed00042f0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 shutdown_connections 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ed4108cf0 0x7f4ed419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ed41083a0 0x7f4ed419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ec00b6580 0x7f4ec00b8a90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 --2- 172.21.15.28:0/499040365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed4102a60 0x7f4ed419b290 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 >> 172.21.15.28:0/499040365 conn(0x7f4ed4079730 msgr2=0x7f4ed41014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 shutdown_connections 2024-09-21T05:38:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.353+0000 7f4edc2bf640 1 -- 172.21.15.28:0/499040365 wait complete. 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.824+0000 7fee3c742640 1 -- 172.21.15.28:0/58495109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee341083a0 msgr2=0x7fee341087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.824+0000 7fee3c742640 1 --2- 172.21.15.28:0/58495109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee341083a0 0x7fee341087b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee24009900 tx=0x7fee2402a410 comp rx=0 tx=0).stop 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 -- 172.21.15.28:0/58495109 shutdown_connections 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 --2- 172.21.15.28:0/58495109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 --2- 172.21.15.28:0/58495109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee341083a0 0x7fee341087b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 --2- 172.21.15.28:0/58495109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee34102a60 0x7fee34102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 -- 172.21.15.28:0/58495109 >> 172.21.15.28:0/58495109 conn(0x7fee34079730 msgr2=0x7fee34079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 -- 172.21.15.28:0/58495109 shutdown_connections 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.825+0000 7fee3c742640 1 -- 172.21.15.28:0/58495109 wait complete. 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 Processor -- start 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 -- start start 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee341083a0 0x7fee3419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee34112960 con 0x7fee34102a60 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee341127e0 con 0x7fee34108cf0 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3c742640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fee34112ae0 con 0x7fee341083a0 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3a4de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3a4de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39772/0 (socket says 172.21.15.28:39772) 2024-09-21T05:38:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3a4de640 1 -- 172.21.15.28:0/4042274386 learned_addr learned my addr 172.21.15.28:0/4042274386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee3acdf640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.826+0000 7fee39cdd640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee341083a0 0x7fee3419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 -- 172.21.15.28:0/4042274386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee341083a0 msgr2=0x7fee3419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee341083a0 0x7fee3419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 -- 172.21.15.28:0/4042274386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 msgr2=0x7fee3419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 -- 172.21.15.28:0/4042274386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee341a0410 con 0x7fee34102a60 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3acdf640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3a4de640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee3000d770 tx=0x7fee3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30017070 con 0x7fee34102a60 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee300054d0 con 0x7fee34102a60 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee341a0700 con 0x7fee34102a60 2024-09-21T05:38:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30004c30 con 0x7fee34102a60 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.827+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee341a6e60 con 0x7fee34102a60 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.828+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee34102e70 con 0x7fee34102a60 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.831+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fee30004e10 con 0x7fee34102a60 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.831+0000 7fee2b7fe640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 0x7fee200b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.832+0000 7fee39cdd640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 0x7fee200b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.832+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fee300e5100 con 0x7fee34102a60 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.832+0000 7fee39cdd640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 0x7fee200b8a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee24000c00 tx=0x7fee24031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.836+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fee300b1970 con 0x7fee34102a60 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 -- 172.21.15.28:0/1606166995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8103b60 msgr2=0x7f50d8103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 --2- 172.21.15.28:0/1606166995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8103b60 0x7f50d8103f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009ca0 tx=0x7f50c802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 -- 172.21.15.28:0/1606166995 shutdown_connections 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 --2- 172.21.15.28:0/1606166995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d8109df0 0x7f50d8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 --2- 172.21.15.28:0/1606166995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d81094a0 0x7f50d81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 --2- 172.21.15.28:0/1606166995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8103b60 0x7f50d8103f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 -- 172.21.15.28:0/1606166995 >> 172.21.15.28:0/1606166995 conn(0x7f50d807b850 msgr2=0x7f50d807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 -- 172.21.15.28:0/1606166995 shutdown_connections 2024-09-21T05:38:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.502+0000 7f50df0e1640 1 -- 172.21.15.28:0/1606166995 wait complete. 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 Processor -- start 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 -- start start 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d8103b60 0x7f50d819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d81094a0 0x7f50d819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8113ab0 con 0x7f50d8109df0 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8113930 con 0x7f50d81094a0 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50df0e1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8113c30 con 0x7f50d8103b60 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50dd67e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50dd67e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39722/0 (socket says 172.21.15.28:39722) 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50dd67e640 1 -- 172.21.15.28:0/2528723881 learned_addr learned my addr 172.21.15.28:0/2528723881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.503+0000 7f50dce7d640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d8103b60 0x7f50d819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50d7fff640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d81094a0 0x7f50d819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 -- 172.21.15.28:0/2528723881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d8103b60 msgr2=0x7f50d819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d8103b60 0x7f50d819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 -- 172.21.15.28:0/2528723881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d81094a0 msgr2=0x7f50d819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d81094a0 0x7f50d819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 -- 172.21.15.28:0/2528723881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d81a0350 con 0x7f50d8109df0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50dd67e640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f50d000d750 tx=0x7f50d000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0017070 con 0x7f50d8109df0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d81a0640 con 0x7f50d8109df0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d00054d0 con 0x7f50d8109df0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0004c60 con 0x7f50d8109df0 2024-09-21T05:38:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.504+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d81a6e60 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.505+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d8103f70 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.507+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f50d0009bd0 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.508+0000 7f50d5ffb640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 0x7f50c40b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.508+0000 7f50dce7d640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 0x7f50c40b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.508+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f50d00e5520 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.508+0000 7f50dce7d640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 0x7f50c40b8b10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50c8005e30 tx=0x7f50c8009360 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.512+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f50d00ad990 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.711+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f50d810bbc0 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.715+0000 7f50d5ffb640 1 -- 172.21.15.28:0/2528723881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v34)=0 v34) ==== 129+0+0 (secure 0 0 0) 0x7f50d00b1d70 con 0x7f50d8109df0 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 msgr2=0x7f50c40b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 0x7f50c40b8b10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50c8005e30 tx=0x7f50c8009360 comp rx=0 tx=0).stop 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 msgr2=0x7f50d819fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f50d000d750 tx=0x7f50d000dc10 comp rx=0 tx=0).stop 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 shutdown_connections 2024-09-21T05:38:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50c40b6600 0x7f50c40b8b10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50d8109df0 0x7f50d819fce0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50d81094a0 0x7f50d819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 --2- 172.21.15.28:0/2528723881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50d8103b60 0x7f50d819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 >> 172.21.15.28:0/2528723881 conn(0x7f50d807b850 msgr2=0x7f50d8107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 shutdown_connections 2024-09-21T05:38:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.718+0000 7f50df0e1640 1 -- 172.21.15.28:0/2528723881 wait complete. 2024-09-21T05:38:42.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.027+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fee341a7030 con 0x7fee34102a60 2024-09-21T05:38:42.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.031+0000 7fee2b7fe640 1 -- 172.21.15.28:0/4042274386 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v36)=0 v36) ==== 120+0+0 (secure 0 0 0) 0x7fee300b68b0 con 0x7fee34102a60 2024-09-21T05:38:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 msgr2=0x7fee200b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 0x7fee200b8a60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee24000c00 tx=0x7fee24031400 comp rx=0 tx=0).stop 2024-09-21T05:38:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 msgr2=0x7fee3419a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee3000d770 tx=0x7fee3000dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 shutdown_connections 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee34108cf0 0x7fee3419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee341083a0 0x7fee3419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fee200b6550 0x7fee200b8a60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 --2- 172.21.15.28:0/4042274386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34102a60 0x7fee3419a4a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 >> 172.21.15.28:0/4042274386 conn(0x7fee34079730 msgr2=0x7fee341019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 shutdown_connections 2024-09-21T05:38:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:42.033+0000 7fee3c742640 1 -- 172.21.15.28:0/4042274386 wait complete. 2024-09-21T05:38:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.603+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/1146245326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc102a60 msgr2=0x7fd9cc102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.603+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/1146245326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc102a60 0x7fd9cc102e70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009900 tx=0x7fd9bc02a410 comp rx=0 tx=0).stop 2024-09-21T05:38:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.603+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/1146245326 shutdown_connections 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.603+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/1146245326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/1146245326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc1083a0 0x7fd9cc1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/1146245326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc102a60 0x7fd9cc102e70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/1146245326 >> 172.21.15.28:0/1146245326 conn(0x7fd9cc079730 msgr2=0x7fd9cc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/1146245326 shutdown_connections 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/1146245326 wait complete. 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 Processor -- start 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.604+0000 7fd9d1c6f640 1 -- start start 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc102a60 0x7fd9cc19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc1083a0 0x7fd9cc19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc112960 con 0x7fd9cc108cf0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1127e0 con 0x7fd9cc102a60 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9d1c6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc112ae0 con 0x7fd9cc1083a0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc102a60 0x7fd9cc19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39744/0 (socket says 172.21.15.28:39744) 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 -- 172.21.15.28:0/763581308 learned_addr learned my addr 172.21.15.28:0/763581308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9c3fff640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc1083a0 0x7fd9cc19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 -- 172.21.15.28:0/763581308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc1083a0 msgr2=0x7fd9cc19b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc1083a0 0x7fd9cc19b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 -- 172.21.15.28:0/763581308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc102a60 msgr2=0x7fd9cc19b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc102a60 0x7fd9cc19b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 -- 172.21.15.28:0/763581308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc1a6470 con 0x7fd9cc108cf0 2024-09-21T05:38:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.605+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400cad0 tx=0x7fd9c400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.606+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400ad60 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.606+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9c4005020 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.606+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc1a6760 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.606+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400f6b0 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.606+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9cc1a6ca0 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.607+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9cc102e70 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.610+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fd9c40051d0 con 0x7fd9cc108cf0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.610+0000 7fd9c97fa640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 0x7fd9b80b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.611+0000 7fd9cb7fe640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 0x7fd9b80b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.611+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd9c40e5a80 con 0x7fd9cc108cf0 2024-09-21T05:38:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.611+0000 7fd9cb7fe640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 0x7fd9b80b89e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000c00 tx=0x7fd9bc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.613+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd9c4014030 con 0x7fd9cc108cf0 2024-09-21T05:38:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.808+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fd9cc02e640 con 0x7fd9cc108cf0 2024-09-21T05:38:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.812+0000 7fd9c97fa640 1 -- 172.21.15.28:0/763581308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v35)=0 v35) ==== 121+0+0 (secure 0 0 0) 0x7fd9c40adf10 con 0x7fd9cc108cf0 2024-09-21T05:38:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 msgr2=0x7fd9b80b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 0x7fd9b80b89e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000c00 tx=0x7fd9bc031400 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 msgr2=0x7fd9cc19bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400cad0 tx=0x7fd9c400ced0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 shutdown_connections 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9b80b64d0 0x7fd9b80b89e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc108cf0 0x7fd9cc19bc70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9cc1083a0 0x7fd9cc19b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 --2- 172.21.15.28:0/763581308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9cc102a60 0x7fd9cc19b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 >> 172.21.15.28:0/763581308 conn(0x7fd9cc079730 msgr2=0x7fd9cc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 shutdown_connections 2024-09-21T05:38:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:41.815+0000 7fd9d1c6f640 1 -- 172.21.15.28:0/763581308 wait complete. 2024-09-21T05:38:42.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:42 smithi028 ceph-mon[31227]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:42.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/499040365' entity='client.admin' 2024-09-21T05:38:42.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2528723881' entity='client.admin' 2024-09-21T05:38:42.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/763581308' entity='client.admin' 2024-09-21T05:38:42.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4042274386' entity='client.admin' 2024-09-21T05:38:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:42 smithi129 ceph-mon[34885]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:42 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/499040365' entity='client.admin' 2024-09-21T05:38:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:42 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2528723881' entity='client.admin' 2024-09-21T05:38:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:42 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/763581308' entity='client.admin' 2024-09-21T05:38:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:42 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4042274386' entity='client.admin' 2024-09-21T05:38:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:42 smithi181 ceph-mon[34819]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:42 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/499040365' entity='client.admin' 2024-09-21T05:38:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:42 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2528723881' entity='client.admin' 2024-09-21T05:38:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:42 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/763581308' entity='client.admin' 2024-09-21T05:38:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:42 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4042274386' entity='client.admin' 2024-09-21T05:38:42.845 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-21T05:38:43.165 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-21T05:38:43.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/2226511951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28101f70 msgr2=0x7f1b28102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/2226511951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28101f70 0x7f1b28102380 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009900 tx=0x7f1b1802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/2226511951 shutdown_connections 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/2226511951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b28109e00 0x7f1b28110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/2226511951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b281094b0 0x7f1b281098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/2226511951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28101f70 0x7f1b28102380 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/2226511951 >> 172.21.15.28:0/2226511951 conn(0x7f1b28077c60 msgr2=0x7f1b280780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/2226511951 shutdown_connections 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.754+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/2226511951 wait complete. 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 Processor -- start 2024-09-21T05:38:43.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 -- start start 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 0x7f1b28102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b281094b0 0x7f1b28103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28113ac0 con 0x7f1b28109e00 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28113940 con 0x7f1b281094b0 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b2dfdd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28113c40 con 0x7f1b28101f70 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39808/0 (socket says 172.21.15.28:39808) 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.755+0000 7f1b27fff640 1 -- 172.21.15.28:0/3902428292 learned_addr learned my addr 172.21.15.28:0/3902428292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b26ffd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b281094b0 0x7f1b28103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b277fe640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 0x7f1b28102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 -- 172.21.15.28:0/3902428292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 msgr2=0x7f1b28102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 0x7f1b28102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 -- 172.21.15.28:0/3902428292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b281094b0 msgr2=0x7f1b28103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b281094b0 0x7f1b28103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 -- 172.21.15.28:0/3902428292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b281a87b0 con 0x7f1b28109e00 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b277fe640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 0x7f1b28102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b27fff640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000eaa0 tx=0x7f1b2000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2000cdb0 con 0x7f1b28109e00 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b281a8aa0 con 0x7f1b28109e00 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.756+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b20005020 con 0x7f1b28109e00 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.757+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200116d0 con 0x7f1b28109e00 2024-09-21T05:38:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.757+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b281a8f90 con 0x7f1b28109e00 2024-09-21T05:38:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.758+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b28102380 con 0x7f1b28109e00 2024-09-21T05:38:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.761+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f1b200051d0 con 0x7f1b28109e00 2024-09-21T05:38:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.761+0000 7f1b24ff9640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 0x7f1b100b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.762+0000 7f1b277fe640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 0x7f1b100b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.762+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1b200e58a0 con 0x7f1b28109e00 2024-09-21T05:38:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.762+0000 7f1b277fe640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 0x7f1b100b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b18000c00 tx=0x7f1b18009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.764+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1b20016030 con 0x7f1b28109e00 2024-09-21T05:38:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.964+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1b28104210 con 0x7f1b28109e00 2024-09-21T05:38:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.968+0000 7f1b24ff9640 1 -- 172.21.15.28:0/3902428292 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v37)=0 v37) ==== 138+0+0 (secure 0 0 0) 0x7f1b28104210 con 0x7f1b28109e00 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 msgr2=0x7f1b100b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 0x7f1b100b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b18000c00 tx=0x7f1b18009680 comp rx=0 tx=0).stop 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 msgr2=0x7f1b28103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000eaa0 tx=0x7f1b2000ef60 comp rx=0 tx=0).stop 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 shutdown_connections 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1b100b64d0 0x7f1b100b89e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b28109e00 0x7f1b28103990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b281094b0 0x7f1b28103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 --2- 172.21.15.28:0/3902428292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b28101f70 0x7f1b28102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 >> 172.21.15.28:0/3902428292 conn(0x7f1b28077c60 msgr2=0x7f1b281075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 shutdown_connections 2024-09-21T05:38:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.971+0000 7f1b2dfdd640 1 -- 172.21.15.28:0/3902428292 wait complete. 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3902428292' entity='client.admin' 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3902428292' entity='client.admin' 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:44.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:44.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3902428292' entity='client.admin' 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:38:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:38:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 -- 172.21.15.28:0/2741630648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8881083a0 msgr2=0x7fc8881087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 --2- 172.21.15.28:0/2741630648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8881083a0 0x7fc8881087b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc878009900 tx=0x7fc87802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 -- 172.21.15.28:0/2741630648 shutdown_connections 2024-09-21T05:38:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 --2- 172.21.15.28:0/2741630648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc888108cf0 0x7fc88810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 --2- 172.21.15.28:0/2741630648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8881083a0 0x7fc8881087b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 --2- 172.21.15.28:0/2741630648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888102a60 0x7fc888102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 -- 172.21.15.28:0/2741630648 >> 172.21.15.28:0/2741630648 conn(0x7fc888079730 msgr2=0x7fc888079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 -- 172.21.15.28:0/2741630648 shutdown_connections 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.663+0000 7fc89070c640 1 -- 172.21.15.28:0/2741630648 wait complete. 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 Processor -- start 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 -- start start 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8881083a0 0x7fc88819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888108cf0 0x7fc88819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8881129b0 con 0x7fc888102a60 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc888112830 con 0x7fc8881083a0 2024-09-21T05:38:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc89070c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc888112b30 con 0x7fc888108cf0 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc88e4a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc88e4a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39800/0 (socket says 172.21.15.28:39800) 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc88e4a8640 1 -- 172.21.15.28:0/3410603032 learned_addr learned my addr 172.21.15.28:0/3410603032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc88dca7640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8881083a0 0x7fc88819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.664+0000 7fc88eca9640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888108cf0 0x7fc88819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 -- 172.21.15.28:0/3410603032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888108cf0 msgr2=0x7fc88819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888108cf0 0x7fc88819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 -- 172.21.15.28:0/3410603032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8881083a0 msgr2=0x7fc88819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8881083a0 0x7fc88819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 -- 172.21.15.28:0/3410603032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8881a6550 con 0x7fc888102a60 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc88e4a8640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc88400d770 tx=0x7fc88400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884017070 con 0x7fc888102a60 2024-09-21T05:38:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8840054d0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8881a6840 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884004c30 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.665+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8881a6d50 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.666+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc888103ce0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.668+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7fc884004dd0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.668+0000 7fc87f7fe640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 0x7fc8740b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.669+0000 7fc88dca7640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 0x7fc8740b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.669+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc8840e5160 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.669+0000 7fc88dca7640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 0x7fc8740b8a90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc878031440 tx=0x7fc878005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.673+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc8840ad5f0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.867+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fc88810aac0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.868+0000 7fc87f7fe640 1 -- 172.21.15.28:0/3410603032 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v36)=0 v36) ==== 106+0+0 (secure 0 0 0) 0x7fc8840b19d0 con 0x7fc888102a60 2024-09-21T05:38:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 msgr2=0x7fc8740b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 0x7fc8740b8a90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc878031440 tx=0x7fc878005a20 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 msgr2=0x7fc88819b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc88400d770 tx=0x7fc88400dc30 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 shutdown_connections 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc888108cf0 0x7fc88819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8881083a0 0x7fc88819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8740b6580 0x7fc8740b8a90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.870+0000 7fc89070c640 1 --2- 172.21.15.28:0/3410603032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc888102a60 0x7fc88819b2c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.871+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 >> 172.21.15.28:0/3410603032 conn(0x7fc888079730 msgr2=0x7fc888104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.871+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 shutdown_connections 2024-09-21T05:38:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:43.871+0000 7fc89070c640 1 -- 172.21.15.28:0/3410603032 wait complete. 2024-09-21T05:38:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:46 smithi028 ceph-mon[31227]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:46 smithi129 ceph-mon[34885]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:46.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:46 smithi181 ceph-mon[34819]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:47.598 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.598 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.598 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.598 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.599 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '42500114-77db-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi028') 2024-09-21T05:38:47.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config dump 2024-09-21T05:38:47.858 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 -- 172.21.15.28:0/2431273352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 msgr2=0x7f65a81087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 --2- 172.21.15.28:0/2431273352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a81087d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f659c009900 tx=0x7f659c02a410 comp rx=0 tx=0).stop 2024-09-21T05:38:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 -- 172.21.15.28:0/2431273352 shutdown_connections 2024-09-21T05:38:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 --2- 172.21.15.28:0/2431273352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 0x7f65a810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 --2- 172.21.15.28:0/2431273352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a81087d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 --2- 172.21.15.28:0/2431273352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 -- 172.21.15.28:0/2431273352 >> 172.21.15.28:0/2431273352 conn(0x7f65a8079740 msgr2=0x7f65a8079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 -- 172.21.15.28:0/2431273352 shutdown_connections 2024-09-21T05:38:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.317+0000 7f65ad914640 1 -- 172.21.15.28:0/2431273352 wait complete. 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.318+0000 7f65ad914640 1 Processor -- start 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.318+0000 7f65ad914640 1 -- start start 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.318+0000 7f65ad914640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.318+0000 7f65ad914640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65ad914640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 0x7f65a819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65ad914640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8113aa0 con 0x7f65a81083c0 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65ad914640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8113920 con 0x7f65a8102a80 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65ad914640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8113c20 con 0x7f65a8108d10 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34538/0 (socket says 172.21.15.28:34538) 2024-09-21T05:38:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 -- 172.21.15.28:0/2036945160 learned_addr learned my addr 172.21.15.28:0/2036945160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a6ffd640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a77fe640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 0x7f65a819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 -- 172.21.15.28:0/2036945160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 msgr2=0x7f65a819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 0x7f65a819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 -- 172.21.15.28:0/2036945160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 msgr2=0x7f65a819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 -- 172.21.15.28:0/2036945160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a81a03e0 con 0x7f65a81083c0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a6ffd640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65a67fc640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f659c032fd0 tx=0x7f659c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.319+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c03f030 con 0x7f65a81083c0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.320+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a81a0670 con 0x7f65a81083c0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.320+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f659c004490 con 0x7f65a81083c0 2024-09-21T05:38:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.320+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a81a6e30 con 0x7f65a81083c0 2024-09-21T05:38:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.320+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c035070 con 0x7f65a81083c0 2024-09-21T05:38:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.321+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a8103d00 con 0x7f65a81083c0 2024-09-21T05:38:48.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.323+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f659c042030 con 0x7f65a81083c0 2024-09-21T05:38:48.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.324+0000 7f65ac912640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 0x7f65900b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:48.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.324+0000 7f65a6ffd640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 0x7f65900b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:48.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.324+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f659c106010 con 0x7f65a81083c0 2024-09-21T05:38:48.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.324+0000 7f65a6ffd640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 0x7f65900b89e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f659800bfd0 tx=0x7f659800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.327+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f659c0d27e0 con 0x7f65a81083c0 2024-09-21T05:38:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.514+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f65a810ab30 con 0x7f65a81083c0 2024-09-21T05:38:48.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:48 smithi028 ceph-mon[31227]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.515+0000 7f65ac912640 1 -- 172.21.15.28:0/2036945160 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+22330 (secure 0 0 0) 0x7f659c0d7720 con 0x7f65a81083c0 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global dev auth_debug true 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global basic container_image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 * 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global dev debug_asserts_on_shutdown true 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global basic log_to_file true 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global basic log_to_journald false 2024-09-21T05:38:48.518 INFO:teuthology.orchestra.run.smithi028.stdout:global basic log_to_stderr false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_allow_pool_delete true 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_clock_drift_allowed 1.000000 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_cluster_log_to_file true 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_max_pg_per_osd 10000 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced mon_warn_on_too_few_osds false 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global dev ms_die_on_bug true 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global dev ms_die_on_old_message true 2024-09-21T05:38:48.519 INFO:teuthology.orchestra.run.smithi028.stdout:global dev ms_die_on_skipped_message false 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced osd_pool_default_erasure_code_profile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced auth_service_ticket_ttl 240.000000 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced debug_mon 20/20 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced debug_ms 1/1 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced debug_paxos 20/20 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_data_avail_warn 5 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_down_mkfs_grace 300 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_mgr_mkfs_grace 240 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon dev mon_osd_prime_pg_temp true 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_osd_reporter_subtree_level osd 2024-09-21T05:38:48.520 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mon advanced public_network 172.21.0.0/20 * 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced cephsqlite_lock_renewal_timeout 900000 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced client_mount_timeout 30 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced debug_cephsqlite 20/20 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced debug_client 20/20 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced debug_mgr 20/20 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced debug_ms 1/1 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/cephadm/container_init True * 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/cephadm/migration_current 7 * 2024-09-21T05:38:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/snap_schedule/allow_m_granularity true 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/snap_schedule/dump_on_update true 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mgr/snap_schedule/log_level debug 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev bdev_debug_aio true 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced bdev_enable_discard true 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced bluestore_allocator bitmap * 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev bluestore_block_size 96636764160 * 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev bluestore_fsck_on_mount true 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced debug_bluefs 1/20 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced debug_bluestore 1/20 2024-09-21T05:38:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced debug_ms 1/1 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced debug_osd 20/20 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced debug_rocksdb 4/10 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_misdirected_ops true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_op_order true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_pg_log_writeout true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_shutdown true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_verify_cached_snaps true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_verify_missing_on_start true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd dev osd_debug_verify_stray_on_activate true 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_failsafe_full_ratio 0.950000 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_heartbeat_grace 60 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_max_omap_entries_per_request 10 2024-09-21T05:38:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_mclock_profile high_recovery_ops 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_memory_target_autotune true 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_objectstore bluestore * 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_queue debug_random * 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_op_queue_cut_off debug_random * 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_recover_clone_overlap true 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_recovery_max_chunk 1048576 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_scrub_load_threshold 5.000000 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_scrub_max_interval 600.000000 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd advanced osd_shutdown_pgref_assert true 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds 20/20 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced debug_ms 2/2 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_bal_merge_size 5 2024-09-21T05:38:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_bal_split_bits 3 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_bal_split_size 100 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_frag true 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_debug_scatterstat true 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds dev mds_verify_scatter true 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced client_mount_timeout 600 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_client 20/20 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced debug_ms 1/1 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T05:38:48.525 INFO:teuthology.orchestra.run.smithi028.stdout:client.rgw advanced rgw_cache_enabled true 2024-09-21T05:38:48.526 INFO:teuthology.orchestra.run.smithi028.stdout:client.rgw advanced rgw_enable_ops_log true 2024-09-21T05:38:48.526 INFO:teuthology.orchestra.run.smithi028.stdout:client.rgw advanced rgw_enable_usage_log true 2024-09-21T05:38:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 msgr2=0x7f65900b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 0x7f65900b89e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f659800bfd0 tx=0x7f659800b390 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 msgr2=0x7f65a819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f659c032fd0 tx=0x7f659c004270 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 shutdown_connections 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65a8108d10 0x7f65a819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65900b64d0 0x7f65900b89e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a81083c0 0x7f65a819aa10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 --2- 172.21.15.28:0/2036945160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65a8102a80 0x7f65a819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 >> 172.21.15.28:0/2036945160 conn(0x7f65a8079740 msgr2=0x7f65a81052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 shutdown_connections 2024-09-21T05:38:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:48.518+0000 7f65ad914640 1 -- 172.21.15.28:0/2036945160 wait complete. 2024-09-21T05:38:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:48 smithi129 ceph-mon[34885]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:48.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:48 smithi181 ceph-mon[34819]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:49.074 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T05:38:49.075 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:38:49.075 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T05:38:49.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T05:38:49.092 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-21 05:37:24.811051474 +0000 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-21 05:32:36.563563314 +0000 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-21 05:32:36.563563314 +0000 2024-09-21T05:38:49.151 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-21 05:32:36.563563314 +0000 2024-09-21T05:38:49.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T05:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-21T05:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-21T05:38:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000113513 s, 4.5 MB/s 2024-09-21T05:38:49.218 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T05:38:49.275 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:49.332 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-21 05:37:24.838050742 +0000 2024-09-21T05:38:49.333 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-21 05:32:36.876554914 +0000 2024-09-21T05:38:49.333 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-21 05:32:36.876554914 +0000 2024-09-21T05:38:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-21 05:32:36.876554914 +0000 2024-09-21T05:38:49.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T05:38:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-21T05:38:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-21T05:38:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.3684e-05 s, 5.5 MB/s 2024-09-21T05:38:49.399 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T05:38:49.457 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-21 05:37:24.865050011 +0000 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-21 05:32:37.196546327 +0000 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-21 05:32:37.196546327 +0000 2024-09-21T05:38:49.513 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-21 05:32:37.196546327 +0000 2024-09-21T05:38:49.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T05:38:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-21T05:38:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-21T05:38:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000111113 s, 4.6 MB/s 2024-09-21T05:38:49.578 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T05:38:49.635 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:49.692 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-21 05:37:24.893049252 +0000 2024-09-21T05:38:49.693 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-21 05:32:37.550536826 +0000 2024-09-21T05:38:49.693 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-21 05:32:37.550536826 +0000 2024-09-21T05:38:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-21 05:32:37.550536826 +0000 2024-09-21T05:38:49.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T05:38:49.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:49 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2036945160' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T05:38:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-21T05:38:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-21T05:38:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000110484 s, 4.6 MB/s 2024-09-21T05:38:49.757 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T05:38:49.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:49 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2036945160' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T05:38:49.814 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T05:38:49.815 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T05:38:49.834 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T05:38:49.835 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-21T05:38:49.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:49 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2036945160' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 05:38:21.008057240 +0000 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 05:32:47.798751950 +0000 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 05:32:47.798751950 +0000 2024-09-21T05:38:49.882 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 05:32:47.798751950 +0000 2024-09-21T05:38:49.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T05:38:49.953 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T05:38:49.953 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T05:38:49.954 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000296492 s, 1.7 MB/s 2024-09-21T05:38:49.955 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T05:38:50.014 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-21T05:38:50.070 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 05:38:21.009057218 +0000 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 05:32:48.127744701 +0000 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 05:32:48.127744701 +0000 2024-09-21T05:38:50.071 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 05:32:48.127744701 +0000 2024-09-21T05:38:50.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T05:38:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T05:38:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T05:38:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000245955 s, 2.1 MB/s 2024-09-21T05:38:50.137 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T05:38:50.195 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-21T05:38:50.252 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T05:38:50.306 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 05:38:21.009057218 +0000 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 05:32:48.447737650 +0000 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 05:32:48.447737650 +0000 2024-09-21T05:38:50.307 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 05:32:48.447737650 +0000 2024-09-21T05:38:50.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T05:38:50.331 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T05:38:50.331 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T05:38:50.331 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000216589 s, 2.4 MB/s 2024-09-21T05:38:50.333 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T05:38:50.389 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-21T05:38:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T05:38:50.445 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 05:38:21.009057218 +0000 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 05:32:48.757730818 +0000 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 05:32:48.757730818 +0000 2024-09-21T05:38:50.446 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 05:32:48.757730818 +0000 2024-09-21T05:38:50.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T05:38:50.509 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T05:38:50.509 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T05:38:50.509 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000206925 s, 2.5 MB/s 2024-09-21T05:38:50.510 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T05:38:50.566 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T05:38:50.566 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T05:38:50.586 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T05:38:50.586 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-21T05:38:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:50 smithi129 ceph-mon[34885]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 05:38:18.726870034 +0000 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 05:32:55.877754723 +0000 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 05:32:55.877754723 +0000 2024-09-21T05:38:50.645 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 05:32:55.877754723 +0000 2024-09-21T05:38:50.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T05:38:50.710 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:50 smithi181 ceph-mon[34819]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:50.714 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T05:38:50.714 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T05:38:50.714 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00029282 s, 1.7 MB/s 2024-09-21T05:38:50.716 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T05:38:50.773 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-21T05:38:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:50 smithi028 ceph-mon[31227]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:50.829 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T05:38:50.829 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:50.829 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-21T05:38:50.829 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:50.829 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:50.830 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 05:38:18.726870034 +0000 2024-09-21T05:38:50.830 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 05:32:56.197747965 +0000 2024-09-21T05:38:50.830 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 05:32:56.197747965 +0000 2024-09-21T05:38:50.830 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 05:32:56.197747965 +0000 2024-09-21T05:38:50.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T05:38:50.895 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T05:38:50.895 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T05:38:50.895 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000217968 s, 2.3 MB/s 2024-09-21T05:38:50.897 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T05:38:50.953 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-21T05:38:51.008 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T05:38:51.008 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 05:38:18.727870013 +0000 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 05:32:56.532740891 +0000 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 05:32:56.532740891 +0000 2024-09-21T05:38:51.009 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 05:32:56.532740891 +0000 2024-09-21T05:38:51.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T05:38:51.072 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T05:38:51.072 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T05:38:51.072 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000222407 s, 2.3 MB/s 2024-09-21T05:38:51.073 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T05:38:51.130 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 05:38:18.727870013 +0000 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 05:32:56.858734008 +0000 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 05:32:56.858734008 +0000 2024-09-21T05:38:51.186 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 05:32:56.858734008 +0000 2024-09-21T05:38:51.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T05:38:51.249 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T05:38:51.249 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T05:38:51.249 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000216531 s, 2.4 MB/s 2024-09-21T05:38:51.250 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T05:38:51.306 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-09-21T05:38:51.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T05:38:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:52.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:52 smithi028 ceph-mon[31227]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:52.689 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:52 smithi129 ceph-mon[34885]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:52 smithi181 ceph-mon[34819]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:53.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:38:53.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-09-21T05:38:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:38:54.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:54 smithi028 ceph-mon[31227]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 -- 172.21.15.28:0/1478854389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a81094c0 msgr2=0x7f46a81098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 --2- 172.21.15.28:0/1478854389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a81094c0 0x7f46a81098d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4698009900 tx=0x7f469802a410 comp rx=0 tx=0).stop 2024-09-21T05:38:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 -- 172.21.15.28:0/1478854389 shutdown_connections 2024-09-21T05:38:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 --2- 172.21.15.28:0/1478854389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a8110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 --2- 172.21.15.28:0/1478854389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a81094c0 0x7f46a81098d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 --2- 172.21.15.28:0/1478854389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a8103b80 0x7f46a8103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 -- 172.21.15.28:0/1478854389 >> 172.21.15.28:0/1478854389 conn(0x7f46a80ff610 msgr2=0x7f46a8101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 -- 172.21.15.28:0/1478854389 shutdown_connections 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 -- 172.21.15.28:0/1478854389 wait complete. 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.689+0000 7f46b0702640 1 Processor -- start 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 -- start start 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a81094c0 0x7f46a819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8113b40 con 0x7f46a8103b80 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46a81139c0 con 0x7f46a8109e10 2024-09-21T05:38:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46b0702640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8113cc0 con 0x7f46a81094c0 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34556/0 (socket says 172.21.15.28:34556) 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 -- 172.21.15.28:0/3368700398 learned_addr learned my addr 172.21.15.28:0/3368700398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46adc9d640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a81094c0 0x7f46a819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46aec9f640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 -- 172.21.15.28:0/3368700398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a81094c0 msgr2=0x7f46a819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a81094c0 0x7f46a819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 -- 172.21.15.28:0/3368700398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 msgr2=0x7f46a819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:38:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46ae49e640 1 -- 172.21.15.28:0/3368700398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a81a0520 con 0x7f46a8103b80 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.690+0000 7f46aec9f640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f46ae49e640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b4a0 tx=0x7f46a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4014070 con 0x7f46a8103b80 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a81a0810 con 0x7f46a8103b80 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46a4010ac0 con 0x7f46a8103b80 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4017c30 con 0x7f46a8103b80 2024-09-21T05:38:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.691+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a81a6f70 con 0x7f46a8103b80 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.692+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a8103f90 con 0x7f46a8103b80 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.693+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f46a4017070 con 0x7f46a8103b80 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.694+0000 7f469f7fe640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 0x7f46940b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.694+0000 7f46adc9d640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 0x7f46940b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.694+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f46a40e4a90 con 0x7f46a8103b80 2024-09-21T05:38:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.695+0000 7f46adc9d640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 0x7f46940b89e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f469802aa00 tx=0x7f4698005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:38:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.698+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46a40ace80 con 0x7f46a8103b80 2024-09-21T05:38:54.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:54 smithi181 ceph-mon[34819]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:54 smithi129 ceph-mon[34885]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:38:54.895+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f46a8063800 con 0x7f46940b64d0 2024-09-21T05:38:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:55 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:38:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:55 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:38:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:55 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:55 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:38:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:55 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:38:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:55 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:55.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:55 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:38:55.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:55 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:38:55.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:55 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:38:56.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:56 smithi028 ceph-mon[31227]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:56.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:56 smithi028 ceph-mon[31227]: from='client.14451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:38:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:56 smithi181 ceph-mon[34819]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:56 smithi181 ceph-mon[34819]: from='client.14451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:38:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:56 smithi129 ceph-mon[34885]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:56 smithi129 ceph-mon[34885]: from='client.14451 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:38:57.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:57 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"} : dispatch 2024-09-21T05:38:57.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:57 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"}]': finished 2024-09-21T05:38:57.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:57 smithi181 ceph-mon[34819]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T05:38:57.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:38:57.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:57 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"} : dispatch 2024-09-21T05:38:57.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:57 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"}]': finished 2024-09-21T05:38:57.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:57 smithi028 ceph-mon[31227]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T05:38:57.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:38:57.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:57 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"} : dispatch 2024-09-21T05:38:57.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:57 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1123638336' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aebf5461-2659-41f2-8c87-f7d8bce2cc0d"}]': finished 2024-09-21T05:38:57.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:57 smithi129 ceph-mon[34885]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T05:38:57.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:38:58.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:58 smithi181 ceph-mon[34819]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:58.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:38:58 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1209523070' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:38:58.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:58 smithi028 ceph-mon[31227]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:58.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:38:58 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1209523070' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:38:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:58 smithi129 ceph-mon[34885]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:38:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:38:58 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1209523070' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:00 smithi181 ceph-mon[34819]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:00 smithi028 ceph-mon[31227]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:00 smithi129 ceph-mon[34885]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:02.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:02 smithi028 ceph-mon[31227]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:02 smithi181 ceph-mon[34819]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:02 smithi129 ceph-mon[34885]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:04.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:04 smithi028 ceph-mon[31227]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:04.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:04 smithi181 ceph-mon[34819]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:04 smithi129 ceph-mon[34885]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:06 smithi028 ceph-mon[31227]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:06.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T05:39:06.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:06 smithi181 ceph-mon[34819]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T05:39:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:06 smithi129 ceph-mon[34885]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T05:39:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:07.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:07 smithi181 ceph-mon[34819]: Deploying daemon osd.0 on smithi028 2024-09-21T05:39:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:07 smithi028 ceph-mon[31227]: Deploying daemon osd.0 on smithi028 2024-09-21T05:39:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:07 smithi129 ceph-mon[34885]: Deploying daemon osd.0 on smithi028 2024-09-21T05:39:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:08 smithi028 ceph-mon[31227]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:08 smithi181 ceph-mon[34819]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:08 smithi129 ceph-mon[34885]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:10.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:10 smithi028 ceph-mon[31227]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:10.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:10 smithi181 ceph-mon[34819]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:10.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:10 smithi129 ceph-mon[34885]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:12.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:12 smithi028 ceph-mon[31227]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:12.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:12.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:12.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:12 smithi181 ceph-mon[34819]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:12 smithi129 ceph-mon[34885]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.130+0000 7f469f7fe640 1 -- 172.21.15.28:0/3368700398 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f46a8063800 con 0x7f46940b64d0 2024-09-21T05:39:16.132 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 msgr2=0x7f46940b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 0x7f46940b89e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f469802aa00 tx=0x7f4698005a20 comp rx=0 tx=0).stop 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 msgr2=0x7f46a819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b4a0 tx=0x7f46a400b960 comp rx=0 tx=0).stop 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 shutdown_connections 2024-09-21T05:39:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a8109e10 0x7f46a819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46a81094c0 0x7f46a819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46940b64d0 0x7f46940b89e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 --2- 172.21.15.28:0/3368700398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8103b80 0x7f46a819a5b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 >> 172.21.15.28:0/3368700398 conn(0x7f46a80ff610 msgr2=0x7f46a8108b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.132+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 shutdown_connections 2024-09-21T05:39:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:39:16.133+0000 7f46b0702640 1 -- 172.21.15.28:0/3368700398 wait complete. 2024-09-21T05:39:16.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:16 smithi028 ceph-mon[31227]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:16.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:16.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.762 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.0.service 2024-09-21T05:39:16.764 INFO:tasks.cephadm:Deploying osd.1 on smithi129 with /dev/vg_nvme/lv_4... 2024-09-21T05:39:16.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T05:39:16.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:16 smithi129 ceph-mon[34885]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:16.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:16.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.787 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:16 smithi181 ceph-mon[34819]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:16.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:16.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:39:18.420 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:39:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:39:18.210+0000 7f77e6a92640 -1 osd.0 0 log_to_monitors true 2024-09-21T05:39:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:18 smithi129 ceph-mon[34885]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:18 smithi129 ceph-mon[34885]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T05:39:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:18 smithi181 ceph-mon[34819]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:18 smithi181 ceph-mon[34819]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T05:39:18.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:18 smithi028 ceph-mon[31227]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:18.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:18 smithi028 ceph-mon[31227]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T05:39:19.268 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:39:19.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-09-21T05:39:19.542 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:39:19.430+0000 7f77e2a14640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T05:39:19.542 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:39:19.434+0000 7f77de812640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:39:19.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-mon[31227]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T05:39:19.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-mon[31227]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T05:39:19.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-mon[31227]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:39:19.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:19.597 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:39:19.719 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:19 smithi129 ceph-mon[34885]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T05:39:19.719 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:19 smithi129 ceph-mon[34885]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T05:39:19.719 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:19 smithi129 ceph-mon[34885]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:39:19.719 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:19.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:19 smithi181 ceph-mon[34819]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T05:39:19.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:19 smithi181 ceph-mon[34819]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T05:39:19.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:19 smithi181 ceph-mon[34819]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:39:19.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 -- 172.21.15.129:0/3472074660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 msgr2=0x7ff120102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 --2- 172.21.15.129:0/3472074660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff120102cd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff110009900 tx=0x7ff11002a410 comp rx=0 tx=0).stop 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 -- 172.21.15.129:0/3472074660 shutdown_connections 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 --2- 172.21.15.129:0/3472074660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108b50 0x7ff12010f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 --2- 172.21.15.129:0/3472074660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108200 0x7ff120108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 --2- 172.21.15.129:0/3472074660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff120102cd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 -- 172.21.15.129:0/3472074660 >> 172.21.15.129:0/3472074660 conn(0x7ff1200fe350 msgr2=0x7ff1201007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:39:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 -- 172.21.15.129:0/3472074660 shutdown_connections 2024-09-21T05:39:20.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.077+0000 7ff11ffff640 1 -- 172.21.15.129:0/3472074660 wait complete. 2024-09-21T05:39:20.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 Processor -- start 2024-09-21T05:39:20.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 -- start start 2024-09-21T05:39:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108200 0x7ff12019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108b50 0x7ff12019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1201127f0 con 0x7ff120108b50 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff120112670 con 0x7ff1201028c0 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.078+0000 7ff11ffff640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff120112970 con 0x7ff120108200 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39404/0 (socket says 172.21.15.129:39404) 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 -- 172.21.15.129:0/4005643677 learned_addr learned my addr 172.21.15.129:0/4005643677 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11e7fc640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108200 0x7ff12019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11f7fe640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108b50 0x7ff12019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 -- 172.21.15.129:0/4005643677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108200 msgr2=0x7ff12019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108200 0x7ff12019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 -- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108b50 msgr2=0x7ff12019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108b50 0x7ff12019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 -- 172.21.15.129:0/4005643677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1201a0220 con 0x7ff1201028c0 2024-09-21T05:39:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff11effd640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff110005940 tx=0x7ff1100042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.079+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11003f030 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.080+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1201a04b0 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.080+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff110003cb0 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.080+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110035070 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.080+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1201a6c70 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.081+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff120102cd0 con 0x7ff1201028c0 2024-09-21T05:39:20.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.082+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7ff110042030 con 0x7ff1201028c0 2024-09-21T05:39:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.083+0000 7ff1249ff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 0x7ff10c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.083+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1702+0+0 (secure 0 0 0) 0x7ff110106310 con 0x7ff1201028c0 2024-09-21T05:39:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.083+0000 7ff11e7fc640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 0x7ff10c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.084+0000 7ff11e7fc640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 0x7ff10c0b89e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff120101ee0 tx=0x7ff10800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:39:20.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.091+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff1100d2a30 con 0x7ff1201028c0 2024-09-21T05:39:20.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:20.284+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff1201070e0 con 0x7ff10c0b64d0 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:20.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:20.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:20.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:20.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:20.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:39:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:20.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:21.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:39:21.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi028 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] boot 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T05:39:21.733 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:21 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi028 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] boot 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T05:39:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:21 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi028 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: osd.0 [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] boot 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T05:39:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:21 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T05:39:22.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:22 smithi181 ceph-mon[34819]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:22.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:22 smithi181 ceph-mon[34819]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T05:39:22.737 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:22 smithi129 ceph-mon[34885]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:22.737 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:22 smithi129 ceph-mon[34885]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T05:39:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:22 smithi028 ceph-mon[31227]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T05:39:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:22 smithi028 ceph-mon[31227]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/1526883910' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"}]': finished 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:23.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:23 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/1449519848' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:23.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/1526883910' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"}]': finished 2024-09-21T05:39:23.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T05:39:23.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:23.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/1449519848' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/1526883910' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"} : dispatch 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced7a484-b9ec-49ce-825a-16be7ba41ec2"}]': finished 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:23 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/1449519848' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:24 smithi181 ceph-mon[34819]: pgmap v34: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:24 smithi028 ceph-mon[31227]: pgmap v34: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:24 smithi129 ceph-mon[34885]: pgmap v34: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:25 smithi181 ceph-mon[34819]: pgmap v35: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:25 smithi028 ceph-mon[31227]: pgmap v35: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:25 smithi129 ceph-mon[34885]: pgmap v35: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:27.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:27 smithi129 ceph-mon[34885]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:28.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:27 smithi181 ceph-mon[34819]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:28.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:27 smithi028 ceph-mon[31227]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:30.015 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:29 smithi129 ceph-mon[34885]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:30.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:29 smithi181 ceph-mon[34819]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:30.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:29 smithi028 ceph-mon[31227]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T05:39:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:31.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T05:39:31.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:31.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T05:39:31.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:31.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:31 smithi129 ceph-mon[34885]: Deploying daemon osd.1 on smithi129 2024-09-21T05:39:31.989 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:31 smithi129 ceph-mon[34885]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:31 smithi181 ceph-mon[34819]: Deploying daemon osd.1 on smithi129 2024-09-21T05:39:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:31 smithi181 ceph-mon[34819]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:31 smithi028 ceph-mon[31227]: Deploying daemon osd.1 on smithi129 2024-09-21T05:39:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:31 smithi028 ceph-mon[31227]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:33.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:33 smithi129 ceph-mon[34885]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:34.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:33 smithi181 ceph-mon[34819]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:33 smithi028 ceph-mon[31227]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:36.012 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:35 smithi129 ceph-mon[34885]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:36.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:35 smithi181 ceph-mon[34819]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:35 smithi028 ceph-mon[31227]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:38.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:37 smithi129 ceph-mon[34885]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:38.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:38.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:38.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:38.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:37 smithi181 ceph-mon[34819]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:37 smithi028 ceph-mon[31227]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:40.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:40.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.748+0000 7ff1249ff640 1 -- 172.21.15.129:0/4005643677 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff1201070e0 con 0x7ff10c0b64d0 2024-09-21T05:39:40.751 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 1 on host 'smithi129' 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 msgr2=0x7ff10c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 0x7ff10c0b89e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff120101ee0 tx=0x7ff10800b390 comp rx=0 tx=0).stop 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 msgr2=0x7ff12019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff110005940 tx=0x7ff1100042f0 comp rx=0 tx=0).stop 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 shutdown_connections 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff10c0b64d0 0x7ff10c0b89e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff120108b50 0x7ff12019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:40.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff120108200 0x7ff12019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:40.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 --2- 172.21.15.129:0/4005643677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201028c0 0x7ff12019a2b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:40.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 >> 172.21.15.129:0/4005643677 conn(0x7ff1200fe350 msgr2=0x7ff12006bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:39:40.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 shutdown_connections 2024-09-21T05:39:40.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:39:40.751+0000 7ff11ffff640 1 -- 172.21.15.129:0/4005643677 wait complete. 2024-09-21T05:39:41.353 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:41 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:41.353 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:41 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.353 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:41 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.380 DEBUG:teuthology.orchestra.run.smithi129:osd.1> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.1.service 2024-09-21T05:39:41.382 INFO:tasks.cephadm:Deploying osd.2 on smithi181 with /dev/vg_nvme/lv_4... 2024-09-21T05:39:41.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T05:39:41.409 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:41 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:41.409 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:41 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.409 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:41 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:41 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:39:41.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:41 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:41 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:41.633 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:39:42.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:42 smithi181 ceph-mon[34819]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:42.434 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:42 smithi129 ceph-mon[34885]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:42.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:42 smithi028 ceph-mon[31227]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:42.688 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:39:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:39:42.576+0000 7f36c20ea640 -1 osd.1 0 log_to_monitors true 2024-09-21T05:39:43.407 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:43 smithi129 ceph-mon[34885]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.407 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:43 smithi129 ceph-mon[34885]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:43 smithi181 ceph-mon[34819]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:43 smithi181 ceph-mon[34819]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:43 smithi028 ceph-mon[31227]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:43 smithi028 ceph-mon[31227]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T05:39:43.912 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:39:43.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-09-21T05:39:44.232 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:39:44.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:44.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:44 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:44.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:44 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:39:44.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:44 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:39:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 -- 172.21.15.181:0/4265104707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 msgr2=0x7f9700107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 --2- 172.21.15.181:0/4265104707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f9700107f80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009900 tx=0x7f96e802a410 comp rx=0 tx=0).stop 2024-09-21T05:39:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 -- 172.21.15.181:0/4265104707 shutdown_connections 2024-09-21T05:39:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 --2- 172.21.15.181:0/4265104707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97001084c0 0x7f970010ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 --2- 172.21.15.181:0/4265104707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f9700107f80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.672+0000 7f97053cf640 1 --2- 172.21.15.181:0/4265104707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9700102230 0x7f9700102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.673+0000 7f97053cf640 1 -- 172.21.15.181:0/4265104707 >> 172.21.15.181:0/4265104707 conn(0x7f9700079e40 msgr2=0x7f970007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.673+0000 7f97053cf640 1 -- 172.21.15.181:0/4265104707 shutdown_connections 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.673+0000 7f97053cf640 1 -- 172.21.15.181:0/4265104707 wait complete. 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.673+0000 7f97053cf640 1 Processor -- start 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.673+0000 7f97053cf640 1 -- start start 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9700102230 0x7f970019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 0x7f970019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9700112a00 con 0x7f9700102230 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9700112880 con 0x7f97001084c0 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f97053cf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9700112b80 con 0x7f9700107b70 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:49620/0 (socket says 172.21.15.181:49620) 2024-09-21T05:39:44.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 -- 172.21.15.181:0/512395264 learned_addr learned my addr 172.21.15.181:0/512395264 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96ff7fe640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 0x7f970019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96feffd640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9700102230 0x7f970019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 -- 172.21.15.181:0/512395264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 msgr2=0x7f970019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 0x7f970019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 -- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9700102230 msgr2=0x7f970019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9700102230 0x7f970019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 -- 172.21.15.181:0/512395264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97001a6570 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96ff7fe640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 0x7f970019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.674+0000 7f96f7fff640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96e802a930 tx=0x7f96e8031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.675+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e803f030 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.675+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96e8004490 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.675+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97001a6800 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.675+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8035070 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.675+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97001a6d30 con 0x7f9700107b70 2024-09-21T05:39:44.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.676+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9700102640 con 0x7f9700107b70 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.678+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7f96e8042030 con 0x7f9700107b70 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.679+0000 7f96fcff9640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 0x7f96ec0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.679+0000 7f96feffd640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 0x7f96ec0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.679+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2244+0+0 (secure 0 0 0) 0x7f96e81060e0 con 0x7f9700107b70 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.679+0000 7f96feffd640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 0x7f96ec0b8a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9700110230 tx=0x7f96f0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:39:44.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.682+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f96e80d25e0 con 0x7f9700107b70 2024-09-21T05:39:44.874 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:39:44.872+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9700063800 con 0x7f96ec0b6550 2024-09-21T05:39:45.364 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:39:45.094+0000 7f36be881640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T05:39:45.364 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:39:45.107+0000 7f36b5e62640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:39:45.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi129 2024-09-21T05:39:45.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T05:39:45.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:45.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:45 smithi129 ceph-mon[34885]: from='osd.1 ' entity='osd.1' 2024-09-21T05:39:45.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi129 2024-09-21T05:39:45.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:45.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:45 smithi181 ceph-mon[34819]: from='osd.1 ' entity='osd.1' 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi129 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:45.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:39:45.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:39:45.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:45.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:45 smithi028 ceph-mon[31227]: from='osd.1 ' entity='osd.1' 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] boot 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T05:39:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] boot 2024-09-21T05:39:46.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T05:39:46.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:39:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:39:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T05:39:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:39:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: osd.1 [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] boot 2024-09-21T05:39:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T05:39:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T05:39:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/838546749' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:47 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:47 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"}]': finished 2024-09-21T05:39:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:47 smithi181 ceph-mon[34819]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T05:39:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:39:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:47 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/838546749' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:47 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:47 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"}]': finished 2024-09-21T05:39:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:47 smithi028 ceph-mon[31227]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T05:39:47.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:39:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:47 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/838546749' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:47 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"} : dispatch 2024-09-21T05:39:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:47 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0efbfaff-07a3-4354-b453-5b1ee7fea224"}]': finished 2024-09-21T05:39:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:47 smithi129 ceph-mon[34885]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T05:39:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:39:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:48 smithi181 ceph-mon[34819]: pgmap v49: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:48 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/2100210592' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:48 smithi028 ceph-mon[31227]: pgmap v49: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:48 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/2100210592' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:48 smithi129 ceph-mon[34885]: pgmap v49: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:48 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/2100210592' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:39:50.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:50 smithi181 ceph-mon[34819]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:50 smithi028 ceph-mon[31227]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:50 smithi129 ceph-mon[34885]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:52 smithi181 ceph-mon[34819]: pgmap v52: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:52 smithi028 ceph-mon[31227]: pgmap v52: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:52 smithi129 ceph-mon[34885]: pgmap v52: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:54.410 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:54 smithi181 ceph-mon[34819]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:54 smithi028 ceph-mon[31227]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:54 smithi129 ceph-mon[34885]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:55.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:55 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T05:39:55.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:55 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:55 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T05:39:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:55 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:55 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T05:39:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:55 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:39:56.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:56 smithi181 ceph-mon[34819]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:56.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:56 smithi181 ceph-mon[34819]: Deploying daemon osd.2 on smithi181 2024-09-21T05:39:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:56 smithi028 ceph-mon[31227]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:56 smithi028 ceph-mon[31227]: Deploying daemon osd.2 on smithi181 2024-09-21T05:39:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:56 smithi129 ceph-mon[34885]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:56 smithi129 ceph-mon[34885]: Deploying daemon osd.2 on smithi181 2024-09-21T05:39:58.437 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:39:58 smithi181 ceph-mon[34819]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:39:58 smithi028 ceph-mon[31227]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:39:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:39:58 smithi129 ceph-mon[34885]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:00.408 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:00 smithi181 ceph-mon[34819]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:00.408 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:00 smithi181 ceph-mon[34819]: overall HEALTH_OK 2024-09-21T05:40:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:00 smithi028 ceph-mon[31227]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:00 smithi028 ceph-mon[31227]: overall HEALTH_OK 2024-09-21T05:40:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:00 smithi129 ceph-mon[34885]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:00 smithi129 ceph-mon[34885]: overall HEALTH_OK 2024-09-21T05:40:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:02 smithi181 ceph-mon[34819]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:02 smithi028 ceph-mon[31227]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:02 smithi129 ceph-mon[34885]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:02.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:02.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:05.337 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 2 on host 'smithi181' 2024-09-21T05:40:05.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.334+0000 7f96fcff9640 1 -- 172.21.15.181:0/512395264 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9700063800 con 0x7f96ec0b6550 2024-09-21T05:40:05.339 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 msgr2=0x7f96ec0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 0x7f96ec0b8a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9700110230 tx=0x7f96f0005f90 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 msgr2=0x7f970019b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96e802a930 tx=0x7f96e8031a20 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 shutdown_connections 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97001084c0 0x7f970019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9700107b70 0x7f970019b7f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96ec0b6550 0x7f96ec0b8a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 --2- 172.21.15.181:0/512395264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9700102230 0x7f970019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 >> 172.21.15.181:0/512395264 conn(0x7f9700079e40 msgr2=0x7f9700105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 shutdown_connections 2024-09-21T05:40:05.340 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:40:05.337+0000 7f97053cf640 1 -- 172.21.15.181:0/512395264 wait complete. 2024-09-21T05:40:05.909 DEBUG:teuthology.orchestra.run.smithi181:osd.2> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.2.service 2024-09-21T05:40:05.911 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-09-21T05:40:05.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T05:40:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:40:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:06 smithi028 ceph-mon[31227]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:06 smithi181 ceph-mon[34819]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:06.563 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:06 smithi129 ceph-mon[34885]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:06.563 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:06.563 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:06.563 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:07.198 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:40:07.040+0000 7f543e4ff640 -1 osd.2 0 log_to_monitors true 2024-09-21T05:40:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:07 smithi028 ceph-mon[31227]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:07 smithi028 ceph-mon[31227]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:07 smithi181 ceph-mon[34819]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:07 smithi181 ceph-mon[34819]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:07 smithi129 ceph-mon[34885]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:07 smithi129 ceph-mon[34885]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:08.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:08 smithi028 ceph-mon[31227]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:08 smithi181 ceph-mon[34819]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: from='osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:08 smithi129 ceph-mon[34885]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:40:08.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:40:08.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-09-21T05:40:08.927 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:40:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.387+0000 7ff3cba96640 1 -- 172.21.15.28:0/4293274788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41094a0 msgr2=0x7ff3c41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.387+0000 7ff3cba96640 1 --2- 172.21.15.28:0/4293274788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41094a0 0x7ff3c41098b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009900 tx=0x7ff3b402a410 comp rx=0 tx=0).stop 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 -- 172.21.15.28:0/4293274788 shutdown_connections 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 --2- 172.21.15.28:0/4293274788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 --2- 172.21.15.28:0/4293274788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41094a0 0x7ff3c41098b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 --2- 172.21.15.28:0/4293274788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c4103b60 0x7ff3c4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 -- 172.21.15.28:0/4293274788 >> 172.21.15.28:0/4293274788 conn(0x7ff3c407b850 msgr2=0x7ff3c407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 -- 172.21.15.28:0/4293274788 shutdown_connections 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 -- 172.21.15.28:0/4293274788 wait complete. 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.388+0000 7ff3cba96640 1 Processor -- start 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 -- start start 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c41094a0 0x7ff3c419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4113ab0 con 0x7ff3c4103b60 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4113930 con 0x7ff3c4109df0 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3cba96640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4113c30 con 0x7ff3c41094a0 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48990/0 (socket says 172.21.15.28:48990) 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 -- 172.21.15.28:0/888436311 learned_addr learned my addr 172.21.15.28:0/888436311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9031640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c41094a0 0x7ff3c419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3ca033640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 -- 172.21.15.28:0/888436311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c41094a0 msgr2=0x7ff3c419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c41094a0 0x7ff3c419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 -- 172.21.15.28:0/888436311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 msgr2=0x7ff3c419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3c9832640 1 -- 172.21.15.28:0/888436311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c41a0370 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.389+0000 7ff3ca033640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3c9832640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9f0 tx=0x7ff3c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cd80 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c00054d0 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c41a0660 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0010690 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.390+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c41a0b20 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.391+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c4103f70 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.393+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 154473+0+0 (secure 0 0 0) 0x7ff3c0004e30 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.393+0000 7ff3baffd640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 0x7ff3b00b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.394+0000 7ff3c9031640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 0x7ff3b00b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.394+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2786+0+0 (secure 0 0 0) 0x7ff3c00e6320 con 0x7ff3c4103b60 2024-09-21T05:40:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.394+0000 7ff3c9031640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 0x7ff3b00b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009230 tx=0x7ff3b4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:40:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.396+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff3c0056710 con 0x7ff3c4103b60 2024-09-21T05:40:09.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:40:09.176+0000 7f543a481640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T05:40:09.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:40:09.320+0000 7f5432277640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:09.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:09.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:09.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:09 smithi181 ceph-mon[34819]: from='osd.2 ' entity='osd.2' 2024-09-21T05:40:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:40:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T05:40:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:09 smithi028 ceph-mon[31227]: from='osd.2 ' entity='osd.2' 2024-09-21T05:40:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:09.594+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff3c4063800 con 0x7ff3b00b6500 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:09.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:09.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:09.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:09.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:09 smithi129 ceph-mon[34885]: from='osd.2 ' entity='osd.2' 2024-09-21T05:40:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:40:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:40:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi181 2024-09-21T05:40:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:10.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi181 to 17088M 2024-09-21T05:40:10.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:10.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:10.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:10.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi181 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi181 to 17088M 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:10.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:10.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi181 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi181 to 17088M 2024-09-21T05:40:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:10.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:10.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:10.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:11.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:11 smithi028 ceph-mon[31227]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:11.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:11 smithi028 ceph-mon[31227]: osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] boot 2024-09-21T05:40:11.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:11 smithi028 ceph-mon[31227]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T05:40:11.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:11 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:11.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:11 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:11 smithi129 ceph-mon[34885]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:11 smithi129 ceph-mon[34885]: osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] boot 2024-09-21T05:40:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:11 smithi129 ceph-mon[34885]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T05:40:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:11 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:11 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:11 smithi181 ceph-mon[34819]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:11 smithi181 ceph-mon[34819]: osd.2 [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] boot 2024-09-21T05:40:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:11 smithi181 ceph-mon[34819]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T05:40:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:11 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T05:40:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:11 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:12.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52772]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52772]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52772]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52775]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.276 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:11 smithi028 sudo[52759]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.276 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:11 smithi028 sudo[52759]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.276 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:11 smithi028 sudo[52759]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.276 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52759]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.276 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52763]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.277 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52763]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.277 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52763]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.277 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:40:12 smithi028 sudo[52763]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43349]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43349]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43352]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.319 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43352]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43355]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43355]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43355]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43358]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.320 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43358]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43855]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43855]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43855]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43858]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.544 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43858]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43861]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43861]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43864]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 sudo[43864]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:12.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"} : dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"}]': finished 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.547 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:12.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"} : dispatch 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"}]': finished 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43358]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 sudo[43358]: pam_unix(sudo:session): session closed for user root 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"} : dispatch 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3012118341' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e76bace-e04c-43a5-95d6-753d39f5adb3"}]': finished 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: osdmap e20: 4 total, 3 up, 4 in 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T05:40:12.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T05:40:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:13.337+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7ff3c00a9d90 con 0x7ff3c4103b60 2024-09-21T05:40:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:13 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:13 smithi129 ceph-mon[34885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:13 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1690483215' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:13 smithi129 ceph-mon[34885]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T05:40:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:13.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:13 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:13.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:13 smithi181 ceph-mon[34819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:13.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:13 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1690483215' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:13.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:13 smithi181 ceph-mon[34819]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T05:40:13.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:13 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T05:40:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:13 smithi028 ceph-mon[31227]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T05:40:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:13 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1690483215' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:13 smithi028 ceph-mon[31227]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T05:40:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:14 smithi129 ceph-mon[34885]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:14 smithi129 ceph-mon[34885]: mgrmap e21: x(active, since 2m), standbys: z, y 2024-09-21T05:40:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:14 smithi181 ceph-mon[34819]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:14 smithi181 ceph-mon[34819]: mgrmap e21: x(active, since 2m), standbys: z, y 2024-09-21T05:40:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:14 smithi028 ceph-mon[31227]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:14 smithi028 ceph-mon[31227]: mgrmap e21: x(active, since 2m), standbys: z, y 2024-09-21T05:40:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:16 smithi129 ceph-mon[34885]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:16.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:16 smithi181 ceph-mon[34819]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:16 smithi028 ceph-mon[31227]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:18 smithi129 ceph-mon[34885]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:18.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:18 smithi028 ceph-mon[31227]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:18.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:18 smithi181 ceph-mon[34819]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:20 smithi129 ceph-mon[34885]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:20.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:20 smithi028 ceph-mon[31227]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:20 smithi181 ceph-mon[34819]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:21.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:21 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T05:40:21.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:21 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:21.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:21 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T05:40:21.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:21 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:21 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T05:40:21.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:21 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:22 smithi129 ceph-mon[34885]: Deploying daemon osd.3 on smithi028 2024-09-21T05:40:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:22 smithi129 ceph-mon[34885]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:22 smithi028 ceph-mon[31227]: Deploying daemon osd.3 on smithi028 2024-09-21T05:40:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:22 smithi028 ceph-mon[31227]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:22.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:22 smithi181 ceph-mon[34819]: Deploying daemon osd.3 on smithi028 2024-09-21T05:40:22.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:22 smithi181 ceph-mon[34819]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:24 smithi129 ceph-mon[34885]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:24 smithi181 ceph-mon[34819]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:24 smithi028 ceph-mon[31227]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:26.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:26 smithi028 ceph-mon[31227]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:26 smithi129 ceph-mon[34885]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:26.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:26 smithi181 ceph-mon[34819]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:27.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:27.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:27.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:27.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:27.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:27.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:27.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:28.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:28 smithi028 ceph-mon[31227]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:28 smithi129 ceph-mon[34885]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:28.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:28 smithi181 ceph-mon[34819]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.170+0000 7ff3baffd640 1 -- 172.21.15.28:0/888436311 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff3c4063800 con 0x7ff3b00b6500 2024-09-21T05:40:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-09-21T05:40:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 msgr2=0x7ff3b00b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 0x7ff3b00b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009230 tx=0x7ff3b4031400 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 msgr2=0x7ff3c419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9f0 tx=0x7ff3c000edf0 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 shutdown_connections 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c4109df0 0x7ff3c419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3c41094a0 0x7ff3c419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff3b00b6500 0x7ff3b00b8a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 --2- 172.21.15.28:0/888436311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103b60 0x7ff3c419a4c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.172+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 >> 172.21.15.28:0/888436311 conn(0x7ff3c407b850 msgr2=0x7ff3c4105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.173+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 shutdown_connections 2024-09-21T05:40:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:40:31.173+0000 7ff3cba96640 1 -- 172.21.15.28:0/888436311 wait complete. 2024-09-21T05:40:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.765 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.3.service 2024-09-21T05:40:31.768 INFO:tasks.cephadm:Deploying osd.4 on smithi129 with /dev/vg_nvme/lv_3... 2024-09-21T05:40:31.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T05:40:31.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:31.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:31.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:32.023 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:40:32.482 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:32 smithi129 ceph-mon[34885]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:32.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:32 smithi181 ceph-mon[34819]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:32 smithi028 ceph-mon[31227]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:33.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:33 smithi028 ceph-mon[31227]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T05:40:33.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:40:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:40:33.325+0000 7ffbc2f8e640 -1 osd.3 0 log_to_monitors true 2024-09-21T05:40:33.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:33 smithi129 ceph-mon[34885]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T05:40:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:33 smithi181 ceph-mon[34819]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T05:40:34.530 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:40:34.387+0000 7ffbbf725640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T05:40:34.530 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:40:34.396+0000 7ffbbad0e640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-mon[31227]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-mon[31227]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T05:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-mon[31227]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T05:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-mon[31227]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:34.570 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:40:34.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-09-21T05:40:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:34 smithi129 ceph-mon[34885]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:34 smithi129 ceph-mon[34885]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T05:40:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:34 smithi129 ceph-mon[34885]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T05:40:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:34 smithi129 ceph-mon[34885]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:40:34.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:34 smithi181 ceph-mon[34819]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:34 smithi181 ceph-mon[34819]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T05:40:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:34 smithi181 ceph-mon[34819]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T05:40:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:34 smithi181 ceph-mon[34819]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:40:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:34.883 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 -- 172.21.15.129:0/3335926582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 msgr2=0x7f576c110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 --2- 172.21.15.129:0/3335926582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c110670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5764009900 tx=0x7f576402a410 comp rx=0 tx=0).stop 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 -- 172.21.15.129:0/3335926582 shutdown_connections 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 --2- 172.21.15.129:0/3335926582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c110670 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 --2- 172.21.15.129:0/3335926582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c1094f0 0x7f576c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 --2- 172.21.15.129:0/3335926582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c103bb0 0x7f576c103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 -- 172.21.15.129:0/3335926582 >> 172.21.15.129:0/3335926582 conn(0x7f576c0ff600 msgr2=0x7f576c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 -- 172.21.15.129:0/3335926582 shutdown_connections 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.312+0000 7f5772ee8640 1 -- 172.21.15.129:0/3335926582 wait complete. 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 Processor -- start 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 -- start start 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 0x7f576c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c1094f0 0x7f576c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f576c113b00 con 0x7f576c103bb0 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f576c113980 con 0x7f576c109e40 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5772ee8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f576c113c80 con 0x7f576c1094f0 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5771485640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5770c84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 0x7f576c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5771485640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43874/0 (socket says 172.21.15.129:43874) 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f5771485640 1 -- 172.21.15.129:0/643977284 learned_addr learned my addr 172.21.15.129:0/643977284 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.313+0000 7f576bfff640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c1094f0 0x7f576c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 -- 172.21.15.129:0/643977284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c1094f0 msgr2=0x7f576c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c1094f0 0x7f576c19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 -- 172.21.15.129:0/643977284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 msgr2=0x7f576c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 -- 172.21.15.129:0/643977284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c1a0390 con 0x7f576c103bb0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5771485640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5770c84640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 0x7f576c19a420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f575c00cad0 tx=0x7f575c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c007df0 con 0x7f576c103bb0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576c1a0680 con 0x7f576c103bb0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575c005020 con 0x7f576c103bb0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c00f6b0 con 0x7f576c103bb0 2024-09-21T05:40:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.314+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576c1a6de0 con 0x7f576c103bb0 2024-09-21T05:40:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.315+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f576c103fc0 con 0x7f576c103bb0 2024-09-21T05:40:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.317+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f575c005480 con 0x7f576c103bb0 2024-09-21T05:40:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.318+0000 7f5769ffb640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 0x7f57580b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:40:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.318+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 3604+0+0 (secure 0 0 0) 0x7f575c0e67e0 con 0x7f576c103bb0 2024-09-21T05:40:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.318+0000 7f576bfff640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 0x7f57580b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:40:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.318+0000 7f576bfff640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 0x7f57580b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f576c102e50 tx=0x7f5754005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:40:35.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.321+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f575c014070 con 0x7f576c103bb0 2024-09-21T05:40:35.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:35.514+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f576c063800 con 0x7f57580b6560 2024-09-21T05:40:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:40:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T05:40:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:35.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:35.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:40:35.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi028 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] boot 2024-09-21T05:40:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T05:40:36.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:36.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:36.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi028 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] boot 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:36.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi028 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: osd.3 [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] boot 2024-09-21T05:40:36.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: osdmap e24: 4 total, 4 up, 4 in 2024-09-21T05:40:36.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T05:40:36.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:40:36.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:40:36.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:37.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:37 smithi129 ceph-mon[34885]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:37.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:37 smithi129 ceph-mon[34885]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T05:40:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:37 smithi181 ceph-mon[34819]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:37.814 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:37 smithi181 ceph-mon[34819]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T05:40:37.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:37 smithi028 ceph-mon[31227]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:40:37.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:37 smithi028 ceph-mon[31227]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: pgmap v85: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/3234939714' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"}]': finished 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T05:40:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:40:38.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/1618487101' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: pgmap v85: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/3234939714' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"}]': finished 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:40:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/1618487101' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: pgmap v85: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/3234939714' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"} : dispatch 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e34f6ec9-ba14-4098-af52-2c147d72cba5"}]': finished 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:40:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/1618487101' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:40:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:40 smithi181 ceph-mon[34819]: pgmap v88: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:40.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:40 smithi028 ceph-mon[31227]: pgmap v88: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:40 smithi129 ceph-mon[34885]: pgmap v88: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:42 smithi181 ceph-mon[34819]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:42.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:42 smithi028 ceph-mon[31227]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:42.834 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:42 smithi129 ceph-mon[34885]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:44 smithi129 ceph-mon[34885]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:40:44.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:44 smithi181 ceph-mon[34819]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:40:44.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:44 smithi028 ceph-mon[31227]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:40:46.722 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:46 smithi129 ceph-mon[34885]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:40:46.722 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T05:40:46.722 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:46 smithi181 ceph-mon[34819]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:40:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T05:40:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:46 smithi028 ceph-mon[31227]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:40:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T05:40:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:47.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:47 smithi129 ceph-mon[34885]: Deploying daemon osd.4 on smithi129 2024-09-21T05:40:47.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:47 smithi181 ceph-mon[34819]: Deploying daemon osd.4 on smithi129 2024-09-21T05:40:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:47 smithi028 ceph-mon[31227]: Deploying daemon osd.4 on smithi129 2024-09-21T05:40:48.704 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:48 smithi129 ceph-mon[34885]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T05:40:48.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:48 smithi181 ceph-mon[34819]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T05:40:48.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:48 smithi028 ceph-mon[31227]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T05:40:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:50 smithi181 ceph-mon[34819]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T05:40:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:50 smithi028 ceph-mon[31227]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T05:40:50.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:50 smithi129 ceph-mon[34885]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T05:40:52.694 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:52 smithi129 ceph-mon[34885]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:52.694 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:52.694 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:52.694 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:52 smithi181 ceph-mon[34819]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:52 smithi028 ceph-mon[31227]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:40:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:54.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:54.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:54.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.184+0000 7f5769ffb640 1 -- 172.21.15.129:0/643977284 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f576c063800 con 0x7f57580b6560 2024-09-21T05:40:56.187 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 4 on host 'smithi129' 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 msgr2=0x7f57580b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 0x7f57580b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f576c102e50 tx=0x7f5754005f90 comp rx=0 tx=0).stop 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 msgr2=0x7f576c19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 0x7f576c19a420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f575c00cad0 tx=0x7f575c00cf90 comp rx=0 tx=0).stop 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 shutdown_connections 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f576c109e40 0x7f576c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f576c1094f0 0x7f576c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f57580b6560 0x7f57580b8a70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 --2- 172.21.15.129:0/643977284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576c103bb0 0x7f576c19a420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:40:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 >> 172.21.15.129:0/643977284 conn(0x7f576c0ff600 msgr2=0x7f576c106a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:40:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 shutdown_connections 2024-09-21T05:40:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:40:56.187+0000 7f5772ee8640 1 -- 172.21.15.129:0/643977284 wait complete. 2024-09-21T05:40:56.745 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:56 smithi129 ceph-mon[34885]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:56.745 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:56.745 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.746 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.773 DEBUG:teuthology.orchestra.run.smithi129:osd.4> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.4.service 2024-09-21T05:40:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:56 smithi181 ceph-mon[34819]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:56 smithi028 ceph-mon[31227]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:40:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:56.815 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-09-21T05:40:56.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T05:40:57.071 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:40:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:58 smithi181 ceph-mon[34819]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:58.754 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:58 smithi129 ceph-mon[34885]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:58.754 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:40:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:40:58.587+0000 7f56cf617640 -1 osd.4 0 log_to_monitors true 2024-09-21T05:40:58.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:58 smithi028 ceph-mon[31227]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:40:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:40:59 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:59.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:40:59 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.618 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:40:59.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-09-21T05:40:59.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T05:40:59.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:40:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:40:59 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:40:59.900 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2234369368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae00757b0 msgr2=0x7f7ae0075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2234369368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae00757b0 0x7f7ae0075bc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0009900 tx=0x7f7ad002a410 comp rx=0 tx=0).stop 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2234369368 shutdown_connections 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2234369368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae0110190 0x7f7ae01125d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2234369368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 0x7f7ae010fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2234369368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae00757b0 0x7f7ae0075bc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2234369368 >> 172.21.15.181:0/2234369368 conn(0x7f7ae007b8d0 msgr2=0x7f7ae007bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:41:00.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2234369368 shutdown_connections 2024-09-21T05:41:00.316 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.312+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2234369368 wait complete. 2024-09-21T05:41:00.316 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 Processor -- start 2024-09-21T05:41:00.316 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 -- start start 2024-09-21T05:41:00.317 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 0x7f7ae019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:00.317 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 0x7f7ae019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:00.317 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:00.317 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae0115ca0 con 0x7f7ae00757b0 2024-09-21T05:41:00.317 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae0115b20 con 0x7f7ae0076100 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.313+0000 7f7ae4b9f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae0115e20 con 0x7f7ae0110190 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:37630/0 (socket says 172.21.15.181:37630) 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 -- 172.21.15.181:0/2026229851 learned_addr learned my addr 172.21.15.181:0/2026229851 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7addd9b640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 0x7f7ae019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ade59c640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 0x7f7ae019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 -- 172.21.15.181:0/2026229851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 msgr2=0x7f7ae019cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 0x7f7ae019cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 -- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 msgr2=0x7f7ae019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 0x7f7ae019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 -- 172.21.15.181:0/2026229851 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae01a25c0 con 0x7f7ae0110190 2024-09-21T05:41:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ade59c640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 0x7f7ae019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7aded9d640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800eaa0 tx=0x7f7ad800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800cdb0 con 0x7f7ae0110190 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae01a28b0 con 0x7f7ae0110190 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad8005020 con 0x7f7ae0110190 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad80116d0 con 0x7f7ae0110190 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.314+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae01a9010 con 0x7f7ae0110190 2024-09-21T05:41:00.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.315+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae0075bc0 con 0x7f7ae0110190 2024-09-21T05:41:00.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.317+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f7ad80051d0 con 0x7f7ae0110190 2024-09-21T05:41:00.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.318+0000 7f7ad77fe640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 0x7f7acc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:00.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.318+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4020+0+0 (secure 0 0 0) 0x7f7ad80e6870 con 0x7f7ae0110190 2024-09-21T05:41:00.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.318+0000 7f7ade59c640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 0x7f7acc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:00.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.318+0000 7f7ade59c640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 0x7f7acc0b8a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0031440 tx=0x7f7ad0009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:00.323 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.321+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7ad8016030 con 0x7f7ae0110190 2024-09-21T05:41:00.513 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:00.511+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7ae0102ee0 con 0x7f7acc0b6560 2024-09-21T05:41:00.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi129 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:00 smithi181 ceph-mon[34819]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi129 2024-09-21T05:41:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T05:41:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T05:41:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T05:41:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:00 smithi028 ceph-mon[31227]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi129 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: from='osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:00 smithi129 ceph-mon[34885]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:01.740 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:01.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:01 smithi181 ceph-mon[34819]: from='osd.4 ' entity='osd.4' 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:01 smithi028 ceph-mon[31227]: from='osd.4 ' entity='osd.4' 2024-09-21T05:41:01.864 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:41:01.372+0000 7f56cb599640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T05:41:01.864 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:41:01.501+0000 7f56c338f640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:41:01.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:41:01.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T05:41:01.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:01.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:01.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:01.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:01.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:01 smithi129 ceph-mon[34885]: from='osd.4 ' entity='osd.4' 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] boot 2024-09-21T05:41:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T05:41:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/401043551' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"}]': finished 2024-09-21T05:41:03.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: osdmap e31: 6 total, 5 up, 6 in 2024-09-21T05:41:03.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:03.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:03 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/3092846177' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] boot 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/401043551' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"}]': finished 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: osdmap e31: 6 total, 5 up, 6 in 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:03 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/3092846177' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: osd.4 [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] boot 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/401043551' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"} : dispatch 2024-09-21T05:41:03.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76455120-2b9e-4fe9-9ae7-3f85cf2a54c9"}]': finished 2024-09-21T05:41:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: osdmap e31: 6 total, 5 up, 6 in 2024-09-21T05:41:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:03 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/3092846177' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:04.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:04 smithi181 ceph-mon[34819]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:04.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:04 smithi028 ceph-mon[31227]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:04 smithi129 ceph-mon[34885]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-21T05:41:05.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:05 smithi129 ceph-mon[34885]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:05.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:05 smithi181 ceph-mon[34819]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:05.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:05 smithi028 ceph-mon[31227]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:08.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:07 smithi181 ceph-mon[34819]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:08.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:07 smithi028 ceph-mon[31227]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:08.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:07 smithi129 ceph-mon[34885]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:10.012 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:09 smithi181 ceph-mon[34819]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:10.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:09 smithi028 ceph-mon[31227]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:09 smithi129 ceph-mon[34885]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:11.914 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:11 smithi181 ceph-mon[34819]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:11.915 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:11 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T05:41:11.915 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:11 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:11.915 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:11 smithi181 ceph-mon[34819]: Deploying daemon osd.5 on smithi181 2024-09-21T05:41:12.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:11 smithi028 ceph-mon[31227]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:12.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:11 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T05:41:12.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:11 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:12.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:11 smithi028 ceph-mon[31227]: Deploying daemon osd.5 on smithi181 2024-09-21T05:41:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:11 smithi129 ceph-mon[34885]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:11 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T05:41:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:11 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:11 smithi129 ceph-mon[34885]: Deploying daemon osd.5 on smithi181 2024-09-21T05:41:14.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:13 smithi181 ceph-mon[34819]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:14.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:13 smithi028 ceph-mon[31227]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:14.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:13 smithi129 ceph-mon[34885]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:16.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:15 smithi181 ceph-mon[34819]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:16.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:15 smithi028 ceph-mon[31227]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:16.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:15 smithi129 ceph-mon[34885]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:17.863 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:17 smithi181 ceph-mon[34819]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:17.863 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:17.863 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:17.863 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:18.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:17 smithi028 ceph-mon[31227]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:18.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:18.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:18.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:18.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:17 smithi129 ceph-mon[34885]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:18.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:18.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:18.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:19 smithi181 ceph-mon[34819]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:20.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:19 smithi028 ceph-mon[31227]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:20.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:19 smithi129 ceph-mon[34885]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:20.368 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-09-21T05:41:20.369 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.366+0000 7f7ad77fe640 1 -- 172.21.15.181:0/2026229851 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7ae0102ee0 con 0x7f7acc0b6560 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 msgr2=0x7f7acc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 0x7f7acc0b8a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0031440 tx=0x7f7ad0009680 comp rx=0 tx=0).stop 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 msgr2=0x7f7ae01a1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800eaa0 tx=0x7f7ad800ef60 comp rx=0 tx=0).stop 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 shutdown_connections 2024-09-21T05:41:20.371 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ae0110190 0x7f7ae01a1e90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae0076100 0x7f7ae019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7acc0b6560 0x7f7acc0b8a70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 --2- 172.21.15.181:0/2026229851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00757b0 0x7f7ae019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 >> 172.21.15.181:0/2026229851 conn(0x7f7ae007b8d0 msgr2=0x7f7ae01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 shutdown_connections 2024-09-21T05:41:20.372 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:41:20.369+0000 7f7ae4b9f640 1 -- 172.21.15.181:0/2026229851 wait complete. 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:20.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.922 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:20 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:20.948 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.5.service 2024-09-21T05:41:20.950 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-09-21T05:41:20.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:21.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:20 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:21.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:21.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:21.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:20 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:41:22.387 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:22 smithi181 ceph-mon[34819]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:22.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:22 smithi028 ceph-mon[31227]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:22 smithi129 ceph-mon[34885]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:23.668 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:41:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:41:23.521+0000 7f4b2ade8640 -1 osd.5 0 log_to_monitors true 2024-09-21T05:41:23.919 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:41:23.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-09-21T05:41:24.203 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:41:24.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:24.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:24.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:24 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/3415837465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24102ae0 msgr2=0x7f1c24102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:24.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/3415837465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24102ae0 0x7f1c24102ef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009900 tx=0x7f1c2002a410 comp rx=0 tx=0).stop 2024-09-21T05:41:24.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/3415837465 shutdown_connections 2024-09-21T05:41:24.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/3415837465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24108d70 0x7f1c2410f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/3415837465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 0x7f1c24108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/3415837465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24102ae0 0x7f1c24102ef0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/3415837465 >> 172.21.15.28:0/3415837465 conn(0x7f1c24079770 msgr2=0x7f1c24079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/3415837465 shutdown_connections 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.672+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/3415837465 wait complete. 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.673+0000 7f1c2b9d0640 1 Processor -- start 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.673+0000 7f1c2b9d0640 1 -- start start 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24102ae0 0x7f1c241994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 0x7f1c24199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c24112a30 con 0x7f1c24108d70 2024-09-21T05:41:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c241128b0 con 0x7f1c24102ae0 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1c24112bb0 con 0x7f1c24108420 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58034/0 (socket says 172.21.15.28:58034) 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 -- 172.21.15.28:0/4285372332 learned_addr learned my addr 172.21.15.28:0/4285372332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2976c640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24102ae0 0x7f1c241994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c28f6b640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 0x7f1c24199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 -- 172.21.15.28:0/4285372332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 msgr2=0x7f1c24199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 0x7f1c24199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 -- 172.21.15.28:0/4285372332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24102ae0 msgr2=0x7f1c241994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24102ae0 0x7f1c241994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 -- 172.21.15.28:0/4285372332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2419f450 con 0x7f1c24108d70 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c29f6d640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00cad0 tx=0x7f1c1c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c00ae20 con 0x7f1c24108d70 2024-09-21T05:41:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c1c005020 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2419f740 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.674+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c00f6b0 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.675+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c241a5ea0 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.676+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c24103d60 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.677+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f1c1c0051d0 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.678+0000 7f1c1a7fc640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 0x7f1c100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.678+0000 7f1c2976c640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 0x7f1c100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.678+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4452+0+0 (secure 0 0 0) 0x7f1c1c0e6520 con 0x7f1c24108d70 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.678+0000 7f1c2976c640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 0x7f1c100b8a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c20031440 tx=0x7f1c20005d50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.682+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1c1c014030 con 0x7f1c24108d70 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:24.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:24.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:24 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:24.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:24.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:24 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:24.882+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1c24063800 con 0x7f1c100b6560 2024-09-21T05:41:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi181 2024-09-21T05:41:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi181 to 8544M 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:25.794 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:41:25.386+0000 7f4b26d6a640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T05:41:25.794 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:41:25.394+0000 7f4b22b68640 -1 osd.5 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:41:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi181 2024-09-21T05:41:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi181 to 8544M 2024-09-21T05:41:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T05:41:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T05:41:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi181 2024-09-21T05:41:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi181 to 8544M 2024-09-21T05:41:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T05:41:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: osdmap e32: 6 total, 5 up, 6 in 2024-09-21T05:41:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:41:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:26.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:41:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T05:41:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] boot 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: osdmap e34: 6 total, 6 up, 6 in 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"} : dispatch 2024-09-21T05:41:27.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"}]': finished 2024-09-21T05:41:27.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: osdmap e36: 7 total, 6 up, 7 in 2024-09-21T05:41:27.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] boot 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: osdmap e34: 6 total, 6 up, 6 in 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T05:41:27.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"} : dispatch 2024-09-21T05:41:27.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"}]': finished 2024-09-21T05:41:27.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: osdmap e36: 7 total, 6 up, 7 in 2024-09-21T05:41:27.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:41:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:41:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: osd.5 [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] boot 2024-09-21T05:41:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: osdmap e34: 6 total, 6 up, 6 in 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: osdmap e35: 6 total, 6 up, 6 in 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"} : dispatch 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2061678375' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a460509-dbe4-4d67-a88a-d45cf3c21227"}]': finished 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: osdmap e36: 7 total, 6 up, 7 in 2024-09-21T05:41:27.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:28.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:28 smithi181 ceph-mon[34819]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:28.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:28 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2346463841' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:28.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:28 smithi028 ceph-mon[31227]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:28.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:28 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2346463841' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:28 smithi129 ceph-mon[34885]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-21T05:41:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:28 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2346463841' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:30.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:30 smithi181 ceph-mon[34819]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:30 smithi028 ceph-mon[31227]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:30 smithi129 ceph-mon[34885]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:32.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:32 smithi028 ceph-mon[31227]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:32.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:32 smithi181 ceph-mon[34819]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:32 smithi129 ceph-mon[34885]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:34.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:34 smithi028 ceph-mon[31227]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:41:34.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:34 smithi129 ceph-mon[34885]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:41:34.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:34 smithi181 ceph-mon[34819]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T05:41:36.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:36 smithi028 ceph-mon[31227]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:41:36.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T05:41:36.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:36 smithi181 ceph-mon[34819]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:41:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T05:41:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:36 smithi129 ceph-mon[34885]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T05:41:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T05:41:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:37.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:37 smithi028 ceph-mon[31227]: Deploying daemon osd.6 on smithi028 2024-09-21T05:41:37.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:37 smithi181 ceph-mon[34819]: Deploying daemon osd.6 on smithi028 2024-09-21T05:41:37.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:37 smithi129 ceph-mon[34885]: Deploying daemon osd.6 on smithi028 2024-09-21T05:41:38.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:38 smithi028 ceph-mon[31227]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T05:41:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:38 smithi181 ceph-mon[34819]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T05:41:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:38 smithi129 ceph-mon[34885]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-21T05:41:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:40 smithi028 ceph-mon[31227]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:41:40.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:40 smithi181 ceph-mon[34819]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:41:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:40 smithi129 ceph-mon[34885]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:41:42.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:42 smithi028 ceph-mon[31227]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:42.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:42.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:42.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:42 smithi181 ceph-mon[34819]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:42 smithi129 ceph-mon[34885]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:44.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:44 smithi181 ceph-mon[34819]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:44.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:44 smithi028 ceph-mon[31227]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:44.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:44 smithi129 ceph-mon[34885]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:45.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:45.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.817+0000 7f1c1a7fc640 1 -- 172.21.15.28:0/4285372332 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1c24063800 con 0x7f1c100b6560 2024-09-21T05:41:45.820 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-09-21T05:41:45.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 msgr2=0x7f1c100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:45.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 0x7f1c100b8a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c20031440 tx=0x7f1c20005d50 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 msgr2=0x7f1c2419ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00cad0 tx=0x7f1c1c00cf90 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 shutdown_connections 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c100b6560 0x7f1c100b8a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c24108d70 0x7f1c2419ed20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c24108420 0x7f1c24199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 --2- 172.21.15.28:0/4285372332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c24102ae0 0x7f1c241994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.819+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 >> 172.21.15.28:0/4285372332 conn(0x7f1c24079770 msgr2=0x7f1c24106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.820+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 shutdown_connections 2024-09-21T05:41:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:41:45.820+0000 7f1c2b9d0640 1 -- 172.21.15.28:0/4285372332 wait complete. 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:46.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.626 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.6.service 2024-09-21T05:41:46.629 INFO:tasks.cephadm:Deploying osd.7 on smithi129 with /dev/vg_nvme/lv_2... 2024-09-21T05:41:46.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T05:41:46.883 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:41:46.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:46.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.997 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:46.997 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:46.997 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:47.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:41:47.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:47.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:47.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:47 smithi028 ceph-mon[31227]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:47.886 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:47 smithi129 ceph-mon[34885]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:48.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:47 smithi181 ceph-mon[34819]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:48.044 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:41:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:41:47.885+0000 7f6282604640 -1 osd.6 0 log_to_monitors true 2024-09-21T05:41:48.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:48 smithi129 ceph-mon[34885]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T05:41:48.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:48 smithi028 ceph-mon[31227]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T05:41:49.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:48 smithi181 ceph-mon[34819]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T05:41:49.145 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:41:49.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-09-21T05:41:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: osdmap e37: 7 total, 6 up, 7 in 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi028 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:49.756 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:49.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 -- 172.21.15.129:0/3277133363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b781025f0 msgr2=0x7f3b78102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:49.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 --2- 172.21.15.129:0/3277133363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b781025f0 0x7f3b78102a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009900 tx=0x7f3b7402a480 comp rx=0 tx=0).stop 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 -- 172.21.15.129:0/3277133363 shutdown_connections 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 --2- 172.21.15.129:0/3277133363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 0x7f3b781125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 --2- 172.21.15.129:0/3277133363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b78102f40 0x7f3b7810fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 --2- 172.21.15.129:0/3277133363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b781025f0 0x7f3b78102a00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 -- 172.21.15.129:0/3277133363 >> 172.21.15.129:0/3277133363 conn(0x7f3b78079740 msgr2=0x7f3b78079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 -- 172.21.15.129:0/3277133363 shutdown_connections 2024-09-21T05:41:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.849+0000 7f3b80224640 1 -- 172.21.15.129:0/3277133363 wait complete. 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 Processor -- start 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 -- start start 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b781025f0 0x7f3b7819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 0x7f3b781a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78115c90 con 0x7f3b781101a0 2024-09-21T05:41:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78115b10 con 0x7f3b78102f40 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b80224640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78115e10 con 0x7f3b781025f0 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b7d7bf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b7d7bf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35936/0 (socket says 172.21.15.129:35936) 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.850+0000 7f3b7d7bf640 1 -- 172.21.15.129:0/626266257 learned_addr learned my addr 172.21.15.129:0/626266257 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7dfc0640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b781025f0 0x7f3b7819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7e7c1640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 0x7f3b781a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 -- 172.21.15.129:0/626266257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b781025f0 msgr2=0x7f3b7819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b781025f0 0x7f3b7819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 -- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 msgr2=0x7f3b781a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 0x7f3b781a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 -- 172.21.15.129:0/626266257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781a2610 con 0x7f3b78102f40 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b7d7bf640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00cd80 con 0x7f3b78102f40 2024-09-21T05:41:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b781a2900 con 0x7f3b78102f40 2024-09-21T05:41:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b6c0054d0 con 0x7f3b78102f40 2024-09-21T05:41:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c010690 con 0x7f3b78102f40 2024-09-21T05:41:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.851+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b781a9060 con 0x7f3b78102f40 2024-09-21T05:41:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.852+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b78102a00 con 0x7f3b78102f40 2024-09-21T05:41:49.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.854+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f3b6c004e50 con 0x7f3b78102f40 2024-09-21T05:41:49.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.855+0000 7f3b6affd640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 0x7f3b640b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:41:49.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.855+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4900+0+0 (secure 0 0 0) 0x7f3b6c0e6c80 con 0x7f3b78102f40 2024-09-21T05:41:49.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.855+0000 7f3b7dfc0640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 0x7f3b640b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:41:49.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.856+0000 7f3b7dfc0640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 0x7f3b640b8a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b74031440 tx=0x7f3b74009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:41:49.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:49.858+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3b6c056710 con 0x7f3b78102f40 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: osdmap e37: 7 total, 6 up, 7 in 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi028 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:50.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:41:50.050+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3b7810a4e0 con 0x7f3b640b6560 2024-09-21T05:41:50.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:41:49.646+0000 7f627e586640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T05:41:50.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:41:49.656+0000 7f627a384640 -1 osd.6 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:41:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T05:41:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: osdmap e37: 7 total, 6 up, 7 in 2024-09-21T05:41:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:41:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi028 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:41:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:50.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:51.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: osdmap e38: 7 total, 6 up, 7 in 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:41:51.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:41:51.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:41:51.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:41:51.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:41:51.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:41:51.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] boot 2024-09-21T05:41:51.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: osdmap e39: 7 total, 7 up, 7 in 2024-09-21T05:41:51.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:51.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:51 smithi129 ceph-mon[34885]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] boot 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: osdmap e39: 7 total, 7 up, 7 in 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:52.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:51 smithi181 ceph-mon[34819]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: osd.6 [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] boot 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: osdmap e39: 7 total, 7 up, 7 in 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T05:41:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:51 smithi028 ceph-mon[31227]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: osdmap e40: 7 total, 7 up, 7 in 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/3690883539' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"}]': finished 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: osdmap e41: 8 total, 7 up, 8 in 2024-09-21T05:41:52.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: osdmap e40: 7 total, 7 up, 7 in 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/3690883539' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"}]': finished 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: osdmap e41: 8 total, 7 up, 8 in 2024-09-21T05:41:52.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:41:53.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: osdmap e40: 7 total, 7 up, 7 in 2024-09-21T05:41:53.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/3690883539' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:53.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"} : dispatch 2024-09-21T05:41:53.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ed1f260-f7ce-4489-80a1-043099519502"}]': finished 2024-09-21T05:41:53.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: osdmap e41: 8 total, 7 up, 8 in 2024-09-21T05:41:53.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:41:53.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:53 smithi181 ceph-mon[34819]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:53.961 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:53 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/1434416675' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:54.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:53 smithi028 ceph-mon[31227]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:54.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:53 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/1434416675' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:54.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:53 smithi129 ceph-mon[34885]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:54.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:53 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/1434416675' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:41:56.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:55 smithi181 ceph-mon[34819]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:56.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:55 smithi028 ceph-mon[31227]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:56.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:55 smithi129 ceph-mon[34885]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:57.817 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:57 smithi129 ceph-mon[34885]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:58.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:57 smithi181 ceph-mon[34819]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:58.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:57 smithi028 ceph-mon[31227]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:41:59.802 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:41:59 smithi129 ceph-mon[34885]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:00.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:41:59 smithi181 ceph-mon[34819]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:00.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:41:59 smithi028 ceph-mon[31227]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:01.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T05:42:01.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:01.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:00 smithi181 ceph-mon[34819]: Deploying daemon osd.7 on smithi129 2024-09-21T05:42:01.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T05:42:01.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:01.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:00 smithi028 ceph-mon[31227]: Deploying daemon osd.7 on smithi129 2024-09-21T05:42:01.091 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T05:42:01.091 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:01.091 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:00 smithi129 ceph-mon[34885]: Deploying daemon osd.7 on smithi129 2024-09-21T05:42:02.011 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:01 smithi129 ceph-mon[34885]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:02.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:01 smithi181 ceph-mon[34819]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:02.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:01 smithi028 ceph-mon[31227]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:04.022 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:03 smithi129 ceph-mon[34885]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:04.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:03 smithi181 ceph-mon[34819]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:03 smithi028 ceph-mon[31227]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:06.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:05 smithi181 ceph-mon[34819]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:05 smithi028 ceph-mon[31227]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:06.072 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:05 smithi129 ceph-mon[34885]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:06.997 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:06.998 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:06.998 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:07.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:07.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:07.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:08.457 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:08 smithi129 ceph-mon[34885]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:08.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:08 smithi181 ceph-mon[34819]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:08 smithi028 ceph-mon[31227]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:09.819 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 7 on host 'smithi129' 2024-09-21T05:42:09.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.817+0000 7f3b6affd640 1 -- 172.21.15.129:0/626266257 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3b7810a4e0 con 0x7f3b640b6560 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 msgr2=0x7f3b640b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 0x7f3b640b8a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b74031440 tx=0x7f3b74009560 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 msgr2=0x7f3b7819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00edf0 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 shutdown_connections 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3b640b6560 0x7f3b640b8a70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b781101a0 0x7f3b781a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b78102f40 0x7f3b7819cc40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 --2- 172.21.15.129:0/626266257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b781025f0 0x7f3b7819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 >> 172.21.15.129:0/626266257 conn(0x7f3b78079740 msgr2=0x7f3b78109dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:42:09.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 shutdown_connections 2024-09-21T05:42:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:42:09.820+0000 7f3b80224640 1 -- 172.21.15.129:0/626266257 wait complete. 2024-09-21T05:42:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:09 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:09 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:09 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:10.397 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.7.service 2024-09-21T05:42:10.399 INFO:tasks.cephadm:Deploying osd.8 on smithi181 with /dev/vg_nvme/lv_2... 2024-09-21T05:42:10.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T05:42:10.657 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:42:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:11 smithi129 ceph-mon[34885]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:12.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:11 smithi181 ceph-mon[34819]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:11 smithi028 ceph-mon[31227]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:12.718 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:42:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:42:12.598+0000 7f74b2bac640 -1 osd.7 0 log_to_monitors true 2024-09-21T05:42:12.931 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:42:12.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-09-21T05:42:13.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:12 smithi129 ceph-mon[34885]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:12 smithi129 ceph-mon[34885]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.217 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:42:13.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:12 smithi181 ceph-mon[34819]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:12 smithi181 ceph-mon[34819]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:12 smithi028 ceph-mon[31227]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:12 smithi028 ceph-mon[31227]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 -- 172.21.15.181:0/1886108499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3181058e0 msgr2=0x7fb31810f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 --2- 172.21.15.181:0/1886108499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3181058e0 0x7fb31810f350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb310009ca0 tx=0x7fb31002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 -- 172.21.15.181:0/1886108499 shutdown_connections 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 --2- 172.21.15.181:0/1886108499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3181058e0 0x7fb31810f350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 --2- 172.21.15.181:0/1886108499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318104f90 0x7fb3181053a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 --2- 172.21.15.181:0/1886108499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb318102f80 0x7fb318103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 -- 172.21.15.181:0/1886108499 >> 172.21.15.181:0/1886108499 conn(0x7fb318077aa0 msgr2=0x7fb318077ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 -- 172.21.15.181:0/1886108499 shutdown_connections 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.707+0000 7fb31eba0640 1 -- 172.21.15.181:0/1886108499 wait complete. 2024-09-21T05:42:13.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 Processor -- start 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 -- start start 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318102f80 0x7fb31819b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3181058e0 0x7fb31819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181127d0 con 0x7fb318102f80 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb318112650 con 0x7fb3181058e0 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31eba0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb318112950 con 0x7fb318104f90 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31d39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31d39d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:35418/0 (socket says 172.21.15.181:35418) 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31d39d640 1 -- 172.21.15.181:0/309604361 learned_addr learned my addr 172.21.15.181:0/309604361 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.708+0000 7fb31db9e640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318102f80 0x7fb31819b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:13.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31e39f640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3181058e0 0x7fb31819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 -- 172.21.15.181:0/309604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3181058e0 msgr2=0x7fb31819baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3181058e0 0x7fb31819baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 -- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318102f80 msgr2=0x7fb31819b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318102f80 0x7fb31819b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 -- 172.21.15.181:0/309604361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3181a62a0 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31d39d640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00d740 tx=0x7fb30c00db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c017070 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb30c010460 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3181a6590 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00f600 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.709+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181a6ad0 con 0x7fb318104f90 2024-09-21T05:42:13.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.710+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb318103390 con 0x7fb318104f90 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.712+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fb30c01a070 con 0x7fb318104f90 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.713+0000 7fb30affd640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 0x7fb3040b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.714+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5316+0+0 (secure 0 0 0) 0x7fb30c0e61b0 con 0x7fb318104f90 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.714+0000 7fb31db9e640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 0x7fb3040b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.714+0000 7fb31db9e640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 0x7fb3040b8a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb3181027a0 tx=0x7fb31400a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:42:13.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.716+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb30c0ad5f0 con 0x7fb318104f90 2024-09-21T05:42:13.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:13.906+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb318063800 con 0x7fb3040b6560 2024-09-21T05:42:14.265 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: osdmap e42: 8 total, 7 up, 8 in 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi129 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.266 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T05:42:14.267 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.267 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:14.267 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:14.267 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T05:42:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: osdmap e42: 8 total, 7 up, 8 in 2024-09-21T05:42:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi129 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T05:42:14.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: osdmap e42: 8 total, 7 up, 8 in 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi129 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:14.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:14.920 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:14.921 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:14 smithi181 ceph-mon[34819]: from='osd.7 ' entity='osd.7' 2024-09-21T05:42:15.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:42:14.801+0000 7f74af343640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T05:42:15.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:42:14.844+0000 7f74a6924640 -1 osd.7 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:42:15.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:42:15.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:15.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:14 smithi129 ceph-mon[34885]: from='osd.7 ' entity='osd.7' 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: osdmap e43: 8 total, 7 up, 8 in 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:15.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:14 smithi028 ceph-mon[31227]: from='osd.7 ' entity='osd.7' 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] boot 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: osdmap e44: 8 total, 8 up, 8 in 2024-09-21T05:42:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] boot 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: osdmap e44: 8 total, 8 up, 8 in 2024-09-21T05:42:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: osd.7 [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] boot 2024-09-21T05:42:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: osdmap e44: 8 total, 8 up, 8 in 2024-09-21T05:42:16.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: osdmap e45: 8 total, 8 up, 8 in 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/241378356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"}]': finished 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: osdmap e46: 9 total, 8 up, 9 in 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:16 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/3037183405' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: osdmap e45: 8 total, 8 up, 8 in 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/241378356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"}]': finished 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: osdmap e46: 9 total, 8 up, 9 in 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:16 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/3037183405' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: osdmap e45: 8 total, 8 up, 8 in 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/241378356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"} : dispatch 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a7581c5-4c13-4d96-82a4-c910a21c857c"}]': finished 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: osdmap e46: 9 total, 8 up, 9 in 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:16 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/3037183405' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:18.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:18 smithi181 ceph-mon[34819]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:18 smithi028 ceph-mon[31227]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:18 smithi129 ceph-mon[34885]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:20.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:20 smithi181 ceph-mon[34819]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:20 smithi028 ceph-mon[31227]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:20 smithi129 ceph-mon[34885]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:22.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:22 smithi181 ceph-mon[34819]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:22.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:22 smithi028 ceph-mon[31227]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:22 smithi129 ceph-mon[34885]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:24.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:24 smithi181 ceph-mon[34819]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T05:42:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:24 smithi028 ceph-mon[31227]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T05:42:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:24 smithi129 ceph-mon[34885]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-21T05:42:25.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T05:42:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T05:42:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:25.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T05:42:25.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:26.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:26 smithi181 ceph-mon[34819]: Deploying daemon osd.8 on smithi181 2024-09-21T05:42:26.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:26 smithi181 ceph-mon[34819]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-21T05:42:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:26 smithi028 ceph-mon[31227]: Deploying daemon osd.8 on smithi181 2024-09-21T05:42:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:26 smithi028 ceph-mon[31227]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-21T05:42:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:26 smithi129 ceph-mon[34885]: Deploying daemon osd.8 on smithi181 2024-09-21T05:42:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:26 smithi129 ceph-mon[34885]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-21T05:42:28.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:28 smithi181 ceph-mon[34819]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T05:42:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:28 smithi028 ceph-mon[31227]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T05:42:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:28 smithi129 ceph-mon[34885]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T05:42:30.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:30 smithi181 ceph-mon[34819]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:30.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:30.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:30.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:30 smithi028 ceph-mon[31227]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:30 smithi129 ceph-mon[34885]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:32.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:32 smithi181 ceph-mon[34819]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:32 smithi028 ceph-mon[31227]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:32 smithi129 ceph-mon[34885]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:33.535 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 8 on host 'smithi181' 2024-09-21T05:42:33.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.533+0000 7fb30affd640 1 -- 172.21.15.181:0/309604361 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb318063800 con 0x7fb3040b6560 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.535+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 msgr2=0x7fb3040b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.535+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 0x7fb3040b8a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb3181027a0 tx=0x7fb31400a2d0 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 msgr2=0x7fb31819b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00d740 tx=0x7fb30c00db40 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 shutdown_connections 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3181058e0 0x7fb31819baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb318104f90 0x7fb31819b560 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb3040b6560 0x7fb3040b8a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 --2- 172.21.15.181:0/309604361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb318102f80 0x7fb31819b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:33.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 >> 172.21.15.181:0/309604361 conn(0x7fb318077aa0 msgr2=0x7fb318106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:42:33.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 shutdown_connections 2024-09-21T05:42:33.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:42:33.536+0000 7fb31eba0640 1 -- 172.21.15.181:0/309604361 wait complete. 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:33 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:33.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:33.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:33 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:33.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:33.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:33 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:34.164 DEBUG:teuthology.orchestra.run.smithi181:osd.8> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.8.service 2024-09-21T05:42:34.167 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-09-21T05:42:34.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T05:42:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:42:35.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:35 smithi028 ceph-mon[31227]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:36.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:35 smithi181 ceph-mon[34819]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:36.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:35 smithi129 ceph-mon[34885]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:36.442 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:42:36.239+0000 7fd031067640 -1 osd.8 0 log_to_monitors true 2024-09-21T05:42:36.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:36.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:36.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:36.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:36.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:36.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:36.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:36 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:42:37.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-09-21T05:42:37.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:36 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:36 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:42:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:42:37.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 -- 172.21.15.28:0/34177377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 msgr2=0x7fc668110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:37.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 --2- 172.21.15.28:0/34177377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc668110320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc658009ca0 tx=0x7fc65802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 -- 172.21.15.28:0/34177377 shutdown_connections 2024-09-21T05:42:37.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 --2- 172.21.15.28:0/34177377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc668110320 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 --2- 172.21.15.28:0/34177377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668105290 0x7fc66810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 --2- 172.21.15.28:0/34177377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668104860 0x7fc668104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 -- 172.21.15.28:0/34177377 >> 172.21.15.28:0/34177377 conn(0x7fc668079740 msgr2=0x7fc668079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:42:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 -- 172.21.15.28:0/34177377 shutdown_connections 2024-09-21T05:42:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.710+0000 7fc66d039640 1 -- 172.21.15.28:0/34177377 wait complete. 2024-09-21T05:42:37.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 Processor -- start 2024-09-21T05:42:37.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 -- start start 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668104860 0x7fc66819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668105290 0x7fc66819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc668113ae0 con 0x7fc66810de10 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc668113960 con 0x7fc668104860 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.711+0000 7fc66d039640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc668113c60 con 0x7fc668105290 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43404/0 (socket says 172.21.15.28:43404) 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 -- 172.21.15.28:0/963480300 learned_addr learned my addr 172.21.15.28:0/963480300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66659c640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668105290 0x7fc66819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc666d9d640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668104860 0x7fc66819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:37.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 -- 172.21.15.28:0/963480300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668105290 msgr2=0x7fc66819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668105290 0x7fc66819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 -- 172.21.15.28:0/963480300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668104860 msgr2=0x7fc66819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668104860 0x7fc66819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 -- 172.21.15.28:0/963480300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6681a03c0 con 0x7fc66810de10 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66759e640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc658009430 tx=0x7fc6580319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658003c30 con 0x7fc66810de10 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc658003dd0 con 0x7fc66810de10 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681a0650 con 0x7fc66810de10 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65802faa0 con 0x7fc66810de10 2024-09-21T05:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.712+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a6e10 con 0x7fc66810de10 2024-09-21T05:42:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.713+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc668104c70 con 0x7fc66810de10 2024-09-21T05:42:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.715+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fc65802fc40 con 0x7fc66810de10 2024-09-21T05:42:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.716+0000 7fc65ffff640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 0x7fc6540b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:42:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.716+0000 7fc666d9d640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 0x7fc6540b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:42:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.716+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5748+0+0 (secure 0 0 0) 0x7fc658107ef0 con 0x7fc66810de10 2024-09-21T05:42:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.717+0000 7fc666d9d640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 0x7fc6540b8a70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc66806c5c0 tx=0x7fc66000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:42:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.720+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc6580cf170 con 0x7fc66810de10 2024-09-21T05:42:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:37.921+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc668101c90 con 0x7fc6540b6560 2024-09-21T05:42:38.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi181 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi181 to 5696M 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: osdmap e47: 9 total, 8 up, 9 in 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:38.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi181 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi181 to 5696M 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: osdmap e47: 9 total, 8 up, 9 in 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi181 2024-09-21T05:42:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi181 to 5696M 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: osdmap e47: 9 total, 8 up, 9 in 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:42:38.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:38.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: osdmap e48: 9 total, 8 up, 9 in 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:38 smithi028 ceph-mon[31227]: from='osd.8 ' entity='osd.8' 2024-09-21T05:42:39.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:42:38.662+0000 7fd02cfe9640 -1 osd.8 0 waiting for initial osdmap 2024-09-21T05:42:39.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:42:38.774+0000 7fd024ddf640 -1 osd.8 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: osdmap e48: 9 total, 8 up, 9 in 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:39.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:39.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:39.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:38 smithi181 ceph-mon[34819]: from='osd.8 ' entity='osd.8' 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: osdmap e48: 9 total, 8 up, 9 in 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:42:39.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:42:39.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:39.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:38 smithi129 ceph-mon[34885]: from='osd.8 ' entity='osd.8' 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: osdmap e49: 9 total, 8 up, 9 in 2024-09-21T05:42:40.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:39 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:42:40.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:42:40.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:40.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: osdmap e49: 9 total, 8 up, 9 in 2024-09-21T05:42:40.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:39 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: osdmap e49: 9 total, 8 up, 9 in 2024-09-21T05:42:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:39 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] boot 2024-09-21T05:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: osdmap e50: 9 total, 9 up, 9 in 2024-09-21T05:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"} : dispatch 2024-09-21T05:42:40.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"}]': finished 2024-09-21T05:42:40.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: osdmap e51: 10 total, 9 up, 10 in 2024-09-21T05:42:40.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:40 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] boot 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: osdmap e50: 9 total, 9 up, 9 in 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"} : dispatch 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"}]': finished 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: osdmap e51: 10 total, 9 up, 10 in 2024-09-21T05:42:41.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:40 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: osd.8 [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] boot 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: osdmap e50: 9 total, 9 up, 9 in 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"} : dispatch 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2731394601' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7"}]': finished 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: osdmap e51: 10 total, 9 up, 10 in 2024-09-21T05:42:41.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:40 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:42.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:41 smithi181 ceph-mon[34819]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:42.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:41 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/31197836' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:42.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:41 smithi181 ceph-mon[34819]: osdmap e52: 10 total, 9 up, 10 in 2024-09-21T05:42:42.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:41 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:41 smithi028 ceph-mon[31227]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:41 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/31197836' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:41 smithi028 ceph-mon[31227]: osdmap e52: 10 total, 9 up, 10 in 2024-09-21T05:42:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:41 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:42.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:41 smithi129 ceph-mon[34885]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-09-21T05:42:42.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:41 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/31197836' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:42:42.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:41 smithi129 ceph-mon[34885]: osdmap e52: 10 total, 9 up, 10 in 2024-09-21T05:42:42.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:41 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:42:44.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:43 smithi181 ceph-mon[34819]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:44.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:43 smithi028 ceph-mon[31227]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:44.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:43 smithi129 ceph-mon[34885]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:45 smithi028 ceph-mon[31227]: pgmap v176: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:46.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:45 smithi181 ceph-mon[34819]: pgmap v176: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:46.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:45 smithi129 ceph-mon[34885]: pgmap v176: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:42:47.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:47 smithi028 ceph-mon[31227]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T05:42:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:47 smithi181 ceph-mon[34819]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T05:42:48.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:47 smithi129 ceph-mon[34885]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T05:42:49.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:48 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T05:42:49.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:48 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:49.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:48 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T05:42:49.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:48 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:49.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:48 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T05:42:49.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:48 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:49 smithi181 ceph-mon[34819]: Deploying daemon osd.9 on smithi028 2024-09-21T05:42:50.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:49 smithi181 ceph-mon[34819]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:42:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:49 smithi028 ceph-mon[31227]: Deploying daemon osd.9 on smithi028 2024-09-21T05:42:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:49 smithi028 ceph-mon[31227]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:42:50.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:49 smithi129 ceph-mon[34885]: Deploying daemon osd.9 on smithi028 2024-09-21T05:42:50.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:49 smithi129 ceph-mon[34885]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:42:52.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:51 smithi028 ceph-mon[31227]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T05:42:52.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:51 smithi181 ceph-mon[34819]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T05:42:52.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:51 smithi129 ceph-mon[34885]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T05:42:54.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:53 smithi181 ceph-mon[34819]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:42:54.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:53 smithi028 ceph-mon[31227]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:42:54.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:53 smithi129 ceph-mon[34885]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-21T05:42:55.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:55.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:55.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:55.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:55.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:55.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:55.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:55.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:55.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:56.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:56 smithi028 ceph-mon[31227]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:56 smithi129 ceph-mon[34885]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:56 smithi181 ceph-mon[34819]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:57.869 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-09-21T05:42:57.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.867+0000 7fc65ffff640 1 -- 172.21.15.28:0/963480300 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc668101c90 con 0x7fc6540b6560 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 msgr2=0x7fc6540b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 0x7fc6540b8a70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc66806c5c0 tx=0x7fc66000b390 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 msgr2=0x7fc66819fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc658009430 tx=0x7fc6580319f0 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 shutdown_connections 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc6540b6560 0x7fc6540b8a70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66810de10 0x7fc66819fc90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc668105290 0x7fc66819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 --2- 172.21.15.28:0/963480300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc668104860 0x7fc66819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 >> 172.21.15.28:0/963480300 conn(0x7fc668079740 msgr2=0x7fc668101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 shutdown_connections 2024-09-21T05:42:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:42:57.870+0000 7fc66d039640 1 -- 172.21.15.28:0/963480300 wait complete. 2024-09-21T05:42:58.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:57 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:57 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:57 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:42:58.541 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.9.service 2024-09-21T05:42:58.543 INFO:tasks.cephadm:Deploying osd.10 on smithi129 with /dev/vg_nvme/lv_1... 2024-09-21T05:42:58.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T05:42:58.802 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:43:00.238 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:42:59 smithi129 ceph-mon[34885]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:00.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:42:59 smithi181 ceph-mon[34819]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:00.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:42:59 smithi028 ceph-mon[31227]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:00.951 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:43:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:43:00.685+0000 7f18f6506640 -1 osd.9 0 log_to_monitors true 2024-09-21T05:43:01.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:00 smithi028 ceph-mon[31227]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T05:43:01.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:00 smithi181 ceph-mon[34819]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T05:43:01.339 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:00 smithi129 ceph-mon[34885]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T05:43:01.342 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T05:43:01.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-09-21T05:43:01.626 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:43:02.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 -- 172.21.15.129:0/2008887990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc108d20 msgr2=0x7f28fc10f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 --2- 172.21.15.129:0/2008887990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc108d20 0x7f28fc10f550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009900 tx=0x7f28e402a410 comp rx=0 tx=0).stop 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 -- 172.21.15.129:0/2008887990 shutdown_connections 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 --2- 172.21.15.129:0/2008887990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc108d20 0x7f28fc10f550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 --2- 172.21.15.129:0/2008887990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc1083d0 0x7f28fc1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 --2- 172.21.15.129:0/2008887990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc102ae0 0x7f28fc102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 -- 172.21.15.129:0/2008887990 >> 172.21.15.129:0/2008887990 conn(0x7f28fc079760 msgr2=0x7f28fc079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 -- 172.21.15.129:0/2008887990 shutdown_connections 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.111+0000 7f2902eb6640 1 -- 172.21.15.129:0/2008887990 wait complete. 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 Processor -- start 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 -- start start 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc102ae0 0x7f28fc1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:02.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 0x7f28fc19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc112990 con 0x7f28fc108d20 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc112810 con 0x7f28fc1083d0 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2902eb6640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc112b10 con 0x7f28fc102ae0 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52950/0 (socket says 172.21.15.129:52950) 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 -- 172.21.15.129:0/1296227273 learned_addr learned my addr 172.21.15.129:0/1296227273 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2901453640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 0x7f28fc19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2900c52640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc102ae0 0x7f28fc1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 -- 172.21.15.129:0/1296227273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc102ae0 msgr2=0x7f28fc1994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc102ae0 0x7f28fc1994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 -- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 msgr2=0x7f28fc19ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 0x7f28fc19ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:02.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f28fbfff640 1 -- 172.21.15.129:0/1296227273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc19f410 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.112+0000 7f2901453640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 0x7f28fc19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f28fbfff640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00ccb0 tx=0x7f28ec00a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec004b20 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28ec004cc0 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00f690 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc19f700 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.113+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1a5e60 con 0x7f28fc1083d0 2024-09-21T05:43:02.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.114+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc103d10 con 0x7f28fc1083d0 2024-09-21T05:43:02.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.116+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f28ec00f8f0 con 0x7f28fc1083d0 2024-09-21T05:43:02.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.117+0000 7f28f9ffb640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 0x7f28e80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:02.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.117+0000 7f2900c52640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 0x7f28e80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:02.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.117+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6196+0+0 (secure 0 0 0) 0x7f28ec0e6f30 con 0x7f28fc1083d0 2024-09-21T05:43:02.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.118+0000 7f2900c52640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 0x7f28e80b8af0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f400b590 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:02.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.121+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28ec014030 con 0x7f28fc1083d0 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: osdmap e53: 10 total, 9 up, 10 in 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi028 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:02.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:01 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: osdmap e53: 10 total, 9 up, 10 in 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi028 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:02.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:01 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:02.343+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f28fc063800 con 0x7f28e80b65e0 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: osdmap e53: 10 total, 9 up, 10 in 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]} : dispatch 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi028 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.345 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:02.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-21T05:43:02.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:02.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:02.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:02.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:01 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:03.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:43:02.773+0000 7f18f2488640 -1 osd.9 0 waiting for initial osdmap 2024-09-21T05:43:03.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:43:02.786+0000 7f18ea27e640 -1 osd.9 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:43:03.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:43:03.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: osdmap e54: 10 total, 9 up, 10 in 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:03.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:02 smithi028 ceph-mon[31227]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' 2024-09-21T05:43:03.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: osdmap e54: 10 total, 9 up, 10 in 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:03.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:02 smithi181 ceph-mon[34819]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: osdmap e54: 10 total, 9 up, 10 in 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:03.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:03.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:02 smithi129 ceph-mon[34885]: from='osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742]' entity='osd.9' 2024-09-21T05:43:04.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:43:04.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:43:04.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:04.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:04.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] boot 2024-09-21T05:43:04.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: osdmap e55: 10 total, 10 up, 10 in 2024-09-21T05:43:04.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] boot 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: osdmap e55: 10 total, 10 up, 10 in 2024-09-21T05:43:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: osd.9 [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] boot 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: osdmap e55: 10 total, 10 up, 10 in 2024-09-21T05:43:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T05:43:05.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:04 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/277973083' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:04 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:04 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"}]': finished 2024-09-21T05:43:05.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:04 smithi181 ceph-mon[34819]: osdmap e56: 11 total, 10 up, 11 in 2024-09-21T05:43:05.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:04 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:05.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:04 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/277973083' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:04 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:04 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"}]': finished 2024-09-21T05:43:05.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:04 smithi028 ceph-mon[31227]: osdmap e56: 11 total, 10 up, 11 in 2024-09-21T05:43:05.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:04 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:05.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:04 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/277973083' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:04 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"} : dispatch 2024-09-21T05:43:05.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:04 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d6e3073-eab4-4ac9-a926-c42fa4e501d0"}]': finished 2024-09-21T05:43:05.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:04 smithi129 ceph-mon[34885]: osdmap e56: 11 total, 10 up, 11 in 2024-09-21T05:43:05.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:04 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:06.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:06 smithi181 ceph-mon[34819]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:06.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:06 smithi181 ceph-mon[34819]: from='client.? 172.21.15.129:0/1859353448' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:06.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:06 smithi028 ceph-mon[31227]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:06.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:06 smithi028 ceph-mon[31227]: from='client.? 172.21.15.129:0/1859353448' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:06.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:06 smithi129 ceph-mon[34885]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:06.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:06 smithi129 ceph-mon[34885]: from='client.? 172.21.15.129:0/1859353448' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:08.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:08 smithi181 ceph-mon[34819]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:08 smithi028 ceph-mon[31227]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:08.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:08 smithi129 ceph-mon[34885]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:10.241 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:10 smithi129 ceph-mon[34885]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:10.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:10 smithi181 ceph-mon[34819]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:10 smithi028 ceph-mon[31227]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:12.207 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:12 smithi129 ceph-mon[34885]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:12 smithi181 ceph-mon[34819]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:12 smithi028 ceph-mon[31227]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:14 smithi181 ceph-mon[34819]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T05:43:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:14 smithi181 ceph-mon[34819]: Deploying daemon osd.10 on smithi129 2024-09-21T05:43:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:14 smithi028 ceph-mon[31227]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T05:43:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:14 smithi028 ceph-mon[31227]: Deploying daemon osd.10 on smithi129 2024-09-21T05:43:14.335 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:14 smithi129 ceph-mon[34885]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:14.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T05:43:14.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:14.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:14 smithi129 ceph-mon[34885]: Deploying daemon osd.10 on smithi129 2024-09-21T05:43:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:16 smithi181 ceph-mon[34819]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:16 smithi028 ceph-mon[31227]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:16 smithi129 ceph-mon[34885]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:18.218 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:18 smithi129 ceph-mon[34885]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:18.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:18 smithi181 ceph-mon[34819]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:18 smithi028 ceph-mon[31227]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:19.290 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:19.290 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:19.290 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:19.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:19.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:19.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:19.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:19.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:19.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:20.134 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:20 smithi129 ceph-mon[34885]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:20.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:20 smithi181 ceph-mon[34819]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:20.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:20 smithi028 ceph-mon[31227]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.251+0000 7f28f9ffb640 1 -- 172.21.15.129:0/1296227273 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f28fc063800 con 0x7f28e80b65e0 2024-09-21T05:43:22.254 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 10 on host 'smithi129' 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 msgr2=0x7f28e80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 0x7f28e80b8af0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f400b590 comp rx=0 tx=0).stop 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 msgr2=0x7f28fc199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00ccb0 tx=0x7f28ec00a620 comp rx=0 tx=0).stop 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 shutdown_connections 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28e80b65e0 0x7f28e80b8af0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28fc108d20 0x7f28fc19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28fc1083d0 0x7f28fc199a40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 --2- 172.21.15.129:0/1296227273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28fc102ae0 0x7f28fc1994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:22.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 >> 172.21.15.129:0/1296227273 conn(0x7f28fc079760 msgr2=0x7f28fc1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:22.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 shutdown_connections 2024-09-21T05:43:22.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-21T05:43:22.254+0000 7f2902eb6640 1 -- 172.21.15.129:0/1296227273 wait complete. 2024-09-21T05:43:22.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.806 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:22.837 DEBUG:teuthology.orchestra.run.smithi129:osd.10> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.10.service 2024-09-21T05:43:22.839 INFO:tasks.cephadm:Deploying osd.11 on smithi181 with /dev/vg_nvme/lv_1... 2024-09-21T05:43:22.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T05:43:23.097 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:43:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:24 smithi129 ceph-mon[34885]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:24 smithi129 ceph-mon[34885]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:24.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:24 smithi129 ceph-mon[34885]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:24.615 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:43:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:43:24.347+0000 7f68bf036640 -1 osd.10 0 log_to_monitors true 2024-09-21T05:43:24.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:24 smithi181 ceph-mon[34819]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:24.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:24 smithi181 ceph-mon[34819]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:24.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:24 smithi181 ceph-mon[34819]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:24 smithi028 ceph-mon[31227]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:24 smithi028 ceph-mon[31227]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:24 smithi028 ceph-mon[31227]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T05:43:25.415 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T05:43:25.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-09-21T05:43:25.693 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: osdmap e57: 11 total, 10 up, 11 in 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi129 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: osdmap e58: 11 total, 10 up, 11 in 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:25.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: osdmap e57: 11 total, 10 up, 11 in 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi129 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: osdmap e58: 11 total, 10 up, 11 in 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:25.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: osdmap e57: 11 total, 10 up, 11 in 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi129 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: osdmap e58: 11 total, 10 up, 11 in 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:25.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 -- 172.21.15.181:0/2863745739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 msgr2=0x7fbb0c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2863745739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c102e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009900 tx=0x7fbafc02a410 comp rx=0 tx=0).stop 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 -- 172.21.15.181:0/2863745739 shutdown_connections 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2863745739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2863745739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2863745739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c102e70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.114+0000 7fbb10b99640 1 -- 172.21.15.181:0/2863745739 >> 172.21.15.181:0/2863745739 conn(0x7fbb0c079730 msgr2=0x7fbb0c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.115+0000 7fbb10b99640 1 -- 172.21.15.181:0/2863745739 shutdown_connections 2024-09-21T05:43:26.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.115+0000 7fbb10b99640 1 -- 172.21.15.181:0/2863745739 wait complete. 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.115+0000 7fbb10b99640 1 Processor -- start 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 -- start start 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c1129b0 con 0x7fbb0c1083a0 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c112830 con 0x7fbb0c108cf0 2024-09-21T05:43:26.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb10b99640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c112b30 con 0x7fbb0c102a60 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c19b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55878/0 (socket says 172.21.15.181:55878) 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0a59c640 1 -- 172.21.15.181:0/2767097819 learned_addr learned my addr 172.21.15.181:0/2767097819 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0ad9d640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb01d9b640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0ad9d640 1 -- 172.21.15.181:0/2767097819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 msgr2=0x7fbb0c19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.116+0000 7fbb0ad9d640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb0ad9d640 1 -- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 msgr2=0x7fbb0c19b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb0ad9d640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c19b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb0ad9d640 1 -- 172.21.15.181:0/2767097819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c1a64c0 con 0x7fbb0c108cf0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb01d9b640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c19b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb0ad9d640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c19bc80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009fd0 tx=0x7fbb0400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0400ce20 con 0x7fbb0c108cf0 2024-09-21T05:43:26.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c1a67b0 con 0x7fbb0c108cf0 2024-09-21T05:43:26.121 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb04005020 con 0x7fbb0c108cf0 2024-09-21T05:43:26.121 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04010690 con 0x7fbb0c108cf0 2024-09-21T05:43:26.121 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.117+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c1a6cf0 con 0x7fbb0c108cf0 2024-09-21T05:43:26.121 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.118+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb0c102e70 con 0x7fbb0c108cf0 2024-09-21T05:43:26.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.120+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fbb040054d0 con 0x7fbb0c108cf0 2024-09-21T05:43:26.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.121+0000 7fbb03fff640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 0x7fbaf80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:26.122 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.121+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6629+0+0 (secure 0 0 0) 0x7fbb040e7460 con 0x7fbb0c108cf0 2024-09-21T05:43:26.123 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.121+0000 7fbb0a59c640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 0x7fbaf80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:26.123 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.121+0000 7fbb0a59c640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 0x7fbaf80b8a70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbafc005ec0 tx=0x7fbafc005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:26.126 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.124+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbb040ae360 con 0x7fbb0c108cf0 2024-09-21T05:43:26.315 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:26.313+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbb0c063800 con 0x7fbaf80b6560 2024-09-21T05:43:26.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='osd.10 ' entity='osd.10' 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: osdmap e59: 11 total, 10 up, 11 in 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='osd.10 ' entity='osd.10' 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: osdmap e59: 11 total, 10 up, 11 in 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:43:26.421+0000 7f68bafb8640 -1 osd.10 0 waiting for initial osdmap 2024-09-21T05:43:26.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:43:26.540+0000 7f68b2dae640 -1 osd.10 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:43:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='osd.10 ' entity='osd.10' 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: osdmap e59: 11 total, 10 up, 11 in 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:27.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:43:27.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:43:27.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:27.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] boot 2024-09-21T05:43:27.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: osdmap e60: 11 total, 11 up, 11 in 2024-09-21T05:43:27.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] boot 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: osdmap e60: 11 total, 11 up, 11 in 2024-09-21T05:43:27.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 308 MiB used, 894 GiB / 894 GiB avail 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: osd.10 [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] boot 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: osdmap e60: 11 total, 11 up, 11 in 2024-09-21T05:43:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: osdmap e61: 11 total, 11 up, 11 in 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/2843750917' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"}]': finished 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: osdmap e62: 12 total, 11 up, 12 in 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:29.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.181:0/1381232010' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: osdmap e61: 11 total, 11 up, 11 in 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/2843750917' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"}]': finished 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: osdmap e62: 12 total, 11 up, 12 in 2024-09-21T05:43:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:29.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:29.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.181:0/1381232010' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: osdmap e61: 11 total, 11 up, 11 in 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/2843750917' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"} : dispatch 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d4e141-618d-41ae-a3e8-4120e69bc428"}]': finished 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: osdmap e62: 12 total, 11 up, 12 in 2024-09-21T05:43:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:29.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:29.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.181:0/1381232010' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T05:43:32.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:31 smithi181 ceph-mon[34819]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:31 smithi028 ceph-mon[31227]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:31 smithi129 ceph-mon[34885]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:34.007 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:33 smithi181 ceph-mon[34819]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T05:43:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:33 smithi028 ceph-mon[31227]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T05:43:34.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:33 smithi129 ceph-mon[34885]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T05:43:35.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:35 smithi129 ceph-mon[34885]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T05:43:35.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:35 smithi181 ceph-mon[34819]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T05:43:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:35 smithi028 ceph-mon[31227]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T05:43:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:37 smithi181 ceph-mon[34819]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:43:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T05:43:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:37 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:37 smithi181 ceph-mon[34819]: Deploying daemon osd.11 on smithi181 2024-09-21T05:43:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:37 smithi028 ceph-mon[31227]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:43:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T05:43:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:37 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:37 smithi028 ceph-mon[31227]: Deploying daemon osd.11 on smithi181 2024-09-21T05:43:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:37 smithi129 ceph-mon[34885]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T05:43:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T05:43:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:37 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:38.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:37 smithi129 ceph-mon[34885]: Deploying daemon osd.11 on smithi181 2024-09-21T05:43:39.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:39 smithi181 ceph-mon[34819]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T05:43:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:39 smithi028 ceph-mon[31227]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T05:43:40.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:39 smithi129 ceph-mon[34885]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T05:43:41.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:41 smithi181 ceph-mon[34819]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:41 smithi028 ceph-mon[31227]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:42.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:41 smithi129 ceph-mon[34885]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:42.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:42 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:43.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:43.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:43.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:42 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:43.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:43.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:43.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:42 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:44.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:43 smithi181 ceph-mon[34819]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:44.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:43 smithi028 ceph-mon[31227]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:44.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:43 smithi129 ceph-mon[34885]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T05:43:45.914 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 11 on host 'smithi181' 2024-09-21T05:43:45.914 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.911+0000 7fbb03fff640 1 -- 172.21.15.181:0/2767097819 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fbb0c063800 con 0x7fbaf80b6560 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 msgr2=0x7fbaf80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 0x7fbaf80b8a70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbafc005ec0 tx=0x7fbafc005a20 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 msgr2=0x7fbb0c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c19bc80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009fd0 tx=0x7fbb0400ef10 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 shutdown_connections 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c108cf0 0x7fbb0c19bc80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbaf80b6560 0x7fbaf80b8a70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c1083a0 0x7fbb0c19b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 --2- 172.21.15.181:0/2767097819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c102a60 0x7fbb0c19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:45.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 >> 172.21.15.181:0/2767097819 conn(0x7fbb0c079730 msgr2=0x7fbb0c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:45.917 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 shutdown_connections 2024-09-21T05:43:45.917 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:45.914+0000 7fbb10b99640 1 -- 172.21.15.181:0/2767097819 wait complete. 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.531 DEBUG:teuthology.orchestra.run.smithi181:osd.11> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.11.service 2024-09-21T05:43:46.533 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-21T05:43:46.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:46.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 -- 172.21.15.28:0/2287271391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 msgr2=0x7fb63c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 --2- 172.21.15.28:0/2287271391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c103f70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009900 tx=0x7fb62c02a410 comp rx=0 tx=0).stop 2024-09-21T05:43:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 -- 172.21.15.28:0/2287271391 shutdown_connections 2024-09-21T05:43:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 --2- 172.21.15.28:0/2287271391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 0x7fb63c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 --2- 172.21.15.28:0/2287271391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 0x7fb63c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 --2- 172.21.15.28:0/2287271391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c103f70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 -- 172.21.15.28:0/2287271391 >> 172.21.15.28:0/2287271391 conn(0x7fb63c07b850 msgr2=0x7fb63c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 -- 172.21.15.28:0/2287271391 shutdown_connections 2024-09-21T05:43:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.370+0000 7fb642758640 1 -- 172.21.15.28:0/2287271391 wait complete. 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 Processor -- start 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 -- start start 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 0x7fb63c19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 0x7fb63c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c113ab0 con 0x7fb63c103b60 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c113930 con 0x7fb63c1094a0 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb642758640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c113c30 con 0x7fb63c109df0 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb63bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb63bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46856/0 (socket says 172.21.15.28:46856) 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.371+0000 7fb63bfff640 1 -- 172.21.15.28:0/3791516196 learned_addr learned my addr 172.21.15.28:0/3791516196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb640cf5640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 0x7fb63c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63b7fe640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 0x7fb63c19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 -- 172.21.15.28:0/3791516196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 msgr2=0x7fb63c19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 0x7fb63c19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 -- 172.21.15.28:0/3791516196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 msgr2=0x7fb63c19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 0x7fb63c19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 -- 172.21.15.28:0/3791516196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c1a0330 con 0x7fb63c103b60 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb63bfff640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005940 tx=0x7fb62c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c03f030 con 0x7fb63c103b60 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb62c003cb0 con 0x7fb63c103b60 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63c1a05c0 con 0x7fb63c103b60 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c035070 con 0x7fb63c103b60 2024-09-21T05:43:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.372+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63c1a0af0 con 0x7fb63c103b60 2024-09-21T05:43:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.373+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63c103f70 con 0x7fb63c103b60 2024-09-21T05:43:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.375+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fb62c042030 con 0x7fb63c103b60 2024-09-21T05:43:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.376+0000 7fb6397fa640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 0x7fb6240b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.376+0000 7fb63b7fe640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 0x7fb6240b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.376+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7024+0+0 (secure 0 0 0) 0x7fb62c1076e0 con 0x7fb63c103b60 2024-09-21T05:43:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.376+0000 7fb63b7fe640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 0x7fb6240b8b20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb630005e10 tx=0x7fb630005da0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.379+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb63c1a0af0 con 0x7fb63c103b60 2024-09-21T05:43:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.567+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb63c19b5b0 con 0x7fb63c103b60 2024-09-21T05:43:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.567+0000 7fb6397fa640 1 -- 172.21.15.28:0/3791516196 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v62) ==== 74+0+133 (secure 0 0 0) 0x7fb63c19b5b0 con 0x7fb63c103b60 2024-09-21T05:43:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:43:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 msgr2=0x7fb6240b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 0x7fb6240b8b20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb630005e10 tx=0x7fb630005da0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 msgr2=0x7fb63c19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005940 tx=0x7fb62c0042f0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 shutdown_connections 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb63c109df0 0x7fb63c19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb63c1094a0 0x7fb63c19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6240b6610 0x7fb6240b8b20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 --2- 172.21.15.28:0/3791516196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63c103b60 0x7fb63c19a480 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 >> 172.21.15.28:0/3791516196 conn(0x7fb63c07b850 msgr2=0x7fb63c107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.569+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 shutdown_connections 2024-09-21T05:43:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:47.570+0000 7fb642758640 1 -- 172.21.15.28:0/3791516196 wait complete. 2024-09-21T05:43:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:48 smithi028 ceph-mon[31227]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:48 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3791516196' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:48.254 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":62,"num_osds":12,"num_up_osds":11,"osd_up_since":1726897407,"num_in_osds":12,"osd_in_since":1726897408,"num_remapped_pgs":0} 2024-09-21T05:43:48.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:48 smithi181 ceph-mon[34819]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:48 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3791516196' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:48 smithi129 ceph-mon[34885]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:48 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3791516196' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:48.934 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:43:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:43:48.717+0000 7fc05a242640 -1 osd.11 0 log_to_monitors true 2024-09-21T05:43:49.226 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:49 smithi181 ceph-mon[34819]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.226 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:49 smithi181 ceph-mon[34819]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T05:43:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:49 smithi028 ceph-mon[31227]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:49 smithi028 ceph-mon[31227]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:49 smithi129 ceph-mon[34885]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:49 smithi129 ceph-mon[34885]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 -- 172.21.15.28:0/3131886786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c8108d10 msgr2=0x7f09c810f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 --2- 172.21.15.28:0/3131886786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c8108d10 0x7f09c810f540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009cc0 tx=0x7f09c002cad0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 -- 172.21.15.28:0/3131886786 shutdown_connections 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 --2- 172.21.15.28:0/3131886786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c8108d10 0x7f09c810f540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 --2- 172.21.15.28:0/3131886786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c81083c0 0x7f09c81087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 --2- 172.21.15.28:0/3131886786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8102a80 0x7f09c8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 -- 172.21.15.28:0/3131886786 >> 172.21.15.28:0/3131886786 conn(0x7f09c8079740 msgr2=0x7f09c8079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 -- 172.21.15.28:0/3131886786 shutdown_connections 2024-09-21T05:43:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.931+0000 7f09cde4a640 1 -- 172.21.15.28:0/3131886786 wait complete. 2024-09-21T05:43:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 Processor -- start 2024-09-21T05:43:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 -- start start 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c8102a80 0x7f09c81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8108d10 0x7f09c819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111880 con 0x7f09c81083c0 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111700 con 0x7f09c8102a80 2024-09-21T05:43:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.932+0000 7f09cde4a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111a00 con 0x7f09c8108d10 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46884/0 (socket says 172.21.15.28:46884) 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 -- 172.21.15.28:0/1604344486 learned_addr learned my addr 172.21.15.28:0/1604344486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09c77fe640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c8102a80 0x7f09c81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09c7fff640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8108d10 0x7f09c819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 -- 172.21.15.28:0/1604344486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8108d10 msgr2=0x7f09c819da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8108d10 0x7f09c819da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 -- 172.21.15.28:0/1604344486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c8102a80 msgr2=0x7f09c81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c8102a80 0x7f09c81981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 -- 172.21.15.28:0/1604344486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c819e160 con 0x7f09c81083c0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09beffd640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f09b000d770 tx=0x7f09b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0017070 con 0x7f09c81083c0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09b0009c30 con 0x7f09c81083c0 2024-09-21T05:43:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b000f620 con 0x7f09c81083c0 2024-09-21T05:43:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c819e450 con 0x7f09c81083c0 2024-09-21T05:43:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.933+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c81a4bb0 con 0x7f09c81083c0 2024-09-21T05:43:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.935+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c8103d00 con 0x7f09c81083c0 2024-09-21T05:43:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.936+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f09b001a070 con 0x7f09c81083c0 2024-09-21T05:43:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.937+0000 7f09c57fa640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 0x7f09b40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.937+0000 7f09c77fe640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 0x7f09b40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.937+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7046+0+0 (secure 0 0 0) 0x7f09b00e6830 con 0x7f09c81083c0 2024-09-21T05:43:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.937+0000 7f09c77fe640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 0x7f09b40b8af0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09b8005d00 tx=0x7f09b800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:49.941+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f09b0009de0 con 0x7f09c81083c0 2024-09-21T05:43:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.132+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f09c810ab30 con 0x7f09c81083c0 2024-09-21T05:43:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.135+0000 7f09c57fa640 1 -- 172.21.15.28:0/1604344486 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v64) ==== 74+0+133 (secure 0 0 0) 0x7f09b00b1990 con 0x7f09c81083c0 2024-09-21T05:43:50.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 msgr2=0x7f09b40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 0x7f09b40b8af0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09b8005d00 tx=0x7f09b800b410 comp rx=0 tx=0).stop 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 msgr2=0x7f09c8198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f09b000d770 tx=0x7f09b000dc30 comp rx=0 tx=0).stop 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 shutdown_connections 2024-09-21T05:43:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c8108d10 0x7f09c819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f09b40b65e0 0x7f09b40b8af0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c81083c0 0x7f09c8198790 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 --2- 172.21.15.28:0/1604344486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c8102a80 0x7f09c81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 >> 172.21.15.28:0/1604344486 conn(0x7f09c8079740 msgr2=0x7f09c8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 shutdown_connections 2024-09-21T05:43:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:50.137+0000 7f09cde4a640 1 -- 172.21.15.28:0/1604344486 wait complete. 2024-09-21T05:43:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: osdmap e63: 12 total, 11 up, 12 in 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: Detected new or changed devices on smithi181 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi181 to 4272M 2024-09-21T05:43:50.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: osdmap e63: 12 total, 11 up, 12 in 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: Detected new or changed devices on smithi181 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi181 to 4272M 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:50.546 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: osdmap e63: 12 total, 11 up, 12 in 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: Detected new or changed devices on smithi181 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:43:50.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi181 to 4272M 2024-09-21T05:43:50.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:50.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:50.616 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:50.938 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":64,"num_osds":12,"num_up_osds":11,"osd_up_since":1726897407,"num_in_osds":12,"osd_in_since":1726897408,"num_remapped_pgs":0} 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: osdmap e64: 12 total, 11 up, 12 in 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1604344486' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:51 smithi028 ceph-mon[31227]: from='osd.11 ' entity='osd.11' 2024-09-21T05:43:51.543 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:43:51.066+0000 7fc0561c4640 -1 osd.11 0 waiting for initial osdmap 2024-09-21T05:43:51.544 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:43:51.168+0000 7fc04dfba640 -1 osd.11 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: osdmap e64: 12 total, 11 up, 12 in 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1604344486' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:51 smithi181 ceph-mon[34819]: from='osd.11 ' entity='osd.11' 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: osdmap e64: 12 total, 11 up, 12 in 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1604344486' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:51 smithi129 ceph-mon[34885]: from='osd.11 ' entity='osd.11' 2024-09-21T05:43:51.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T05:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:52.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:52 smithi028 ceph-mon[31227]: purged_snaps scrub starts 2024-09-21T05:43:52.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:52 smithi028 ceph-mon[31227]: purged_snaps scrub ok 2024-09-21T05:43:52.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:52 smithi028 ceph-mon[31227]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:52.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:52.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:52 smithi181 ceph-mon[34819]: purged_snaps scrub starts 2024-09-21T05:43:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:52 smithi181 ceph-mon[34819]: purged_snaps scrub ok 2024-09-21T05:43:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:52 smithi181 ceph-mon[34819]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:52 smithi129 ceph-mon[34885]: purged_snaps scrub starts 2024-09-21T05:43:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:52 smithi129 ceph-mon[34885]: purged_snaps scrub ok 2024-09-21T05:43:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:52 smithi129 ceph-mon[34885]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 -- 172.21.15.28:0/3529774505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 msgr2=0x7f18d81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 --2- 172.21.15.28:0/3529774505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d81087b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009900 tx=0x7f18cc02a410 comp rx=0 tx=0).stop 2024-09-21T05:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 -- 172.21.15.28:0/3529774505 shutdown_connections 2024-09-21T05:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 --2- 172.21.15.28:0/3529774505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 0x7f18d810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 --2- 172.21.15.28:0/3529774505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d81087b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 --2- 172.21.15.28:0/3529774505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 0x7f18d8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 -- 172.21.15.28:0/3529774505 >> 172.21.15.28:0/3529774505 conn(0x7f18d8079730 msgr2=0x7f18d8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 -- 172.21.15.28:0/3529774505 shutdown_connections 2024-09-21T05:43:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 -- 172.21.15.28:0/3529774505 wait complete. 2024-09-21T05:43:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.632+0000 7f18dd88c640 1 Processor -- start 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 -- start start 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 0x7f18d819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 0x7f18d819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18d8112960 con 0x7f18d81083a0 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18d81127e0 con 0x7f18d8102a60 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18dd88c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f18d8112ae0 con 0x7f18d8108cf0 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46896/0 (socket says 172.21.15.28:46896) 2024-09-21T05:43:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 -- 172.21.15.28:0/1320977922 learned_addr learned my addr 172.21.15.28:0/1320977922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d6ffd640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 0x7f18d819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d77fe640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 0x7f18d819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 -- 172.21.15.28:0/1320977922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 msgr2=0x7f18d819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 0x7f18d819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 -- 172.21.15.28:0/1320977922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 msgr2=0x7f18d819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 0x7f18d819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.633+0000 7f18d67fc640 1 -- 172.21.15.28:0/1320977922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d81a03d0 con 0x7f18d81083a0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18d67fc640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18cc032fd0 tx=0x7f18cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc03f030 con 0x7f18d81083a0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d81a0660 con 0x7f18d81083a0 2024-09-21T05:43:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18cc004490 con 0x7f18d81083a0 2024-09-21T05:43:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d81a6e20 con 0x7f18d81083a0 2024-09-21T05:43:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.634+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc035070 con 0x7f18d81083a0 2024-09-21T05:43:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.635+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d8102e70 con 0x7f18d81083a0 2024-09-21T05:43:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.637+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f18cc042030 con 0x7f18d81083a0 2024-09-21T05:43:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.637+0000 7f18dc88a640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 0x7f18c00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:52.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.637+0000 7f18d6ffd640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 0x7f18c00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:52.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.637+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7342+0+0 (secure 0 0 0) 0x7f18cc107860 con 0x7f18d81083a0 2024-09-21T05:43:52.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.638+0000 7f18d6ffd640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 0x7f18c00b8a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18c8005d00 tx=0x7f18c800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:52.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.641+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f18cc0d28a0 con 0x7f18d81083a0 2024-09-21T05:43:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.827+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f18d819b590 con 0x7f18d81083a0 2024-09-21T05:43:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.827+0000 7f18dc88a640 1 -- 172.21.15.28:0/1320977922 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v65) ==== 74+0+133 (secure 0 0 0) 0x7f18d819b590 con 0x7f18d81083a0 2024-09-21T05:43:52.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:43:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.829+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 msgr2=0x7f18c00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.829+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 0x7f18c00b8a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18c8005d00 tx=0x7f18c800b410 comp rx=0 tx=0).stop 2024-09-21T05:43:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.829+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 msgr2=0x7f18d819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.829+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18cc032fd0 tx=0x7f18cc004270 comp rx=0 tx=0).stop 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 shutdown_connections 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18d8108cf0 0x7f18d819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f18c00b6560 0x7f18c00b8a70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d81083a0 0x7f18d819aa00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 --2- 172.21.15.28:0/1320977922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18d8102a60 0x7f18d819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 >> 172.21.15.28:0/1320977922 conn(0x7f18d8079730 msgr2=0x7f18d806b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 shutdown_connections 2024-09-21T05:43:52.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:52.830+0000 7f18dd88c640 1 -- 172.21.15.28:0/1320977922 wait complete. 2024-09-21T05:43:53.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:53 smithi181 ceph-mon[34819]: osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] boot 2024-09-21T05:43:53.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:53 smithi181 ceph-mon[34819]: osdmap e65: 12 total, 12 up, 12 in 2024-09-21T05:43:53.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:53 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1320977922' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:53 smithi028 ceph-mon[31227]: osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] boot 2024-09-21T05:43:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:53 smithi028 ceph-mon[31227]: osdmap e65: 12 total, 12 up, 12 in 2024-09-21T05:43:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:53 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1320977922' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:53.477 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"num_osds":12,"num_up_osds":12,"osd_up_since":1726897432,"num_in_osds":12,"osd_in_since":1726897408,"num_remapped_pgs":0} 2024-09-21T05:43:53.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:43:53.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:53 smithi129 ceph-mon[34885]: osd.11 [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] boot 2024-09-21T05:43:53.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:53 smithi129 ceph-mon[34885]: osdmap e65: 12 total, 12 up, 12 in 2024-09-21T05:43:53.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T05:43:53.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:53 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1320977922' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T05:43:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.146+0000 7f174beb9640 1 -- 172.21.15.28:0/1428087068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744110200 msgr2=0x7f1744112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.146+0000 7f174beb9640 1 --2- 172.21.15.28:0/1428087068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744110200 0x7f1744112620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f173c009ca0 tx=0x7f173c02a950 comp rx=0 tx=0).stop 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 -- 172.21.15.28:0/1428087068 shutdown_connections 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 --2- 172.21.15.28:0/1428087068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744110200 0x7f1744112620 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 --2- 172.21.15.28:0/1428087068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102fa0 0x7f174410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 --2- 172.21.15.28:0/1428087068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744102650 0x7f1744102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 -- 172.21.15.28:0/1428087068 >> 172.21.15.28:0/1428087068 conn(0x7f1744079760 msgr2=0x7f1744079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 -- 172.21.15.28:0/1428087068 shutdown_connections 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 -- 172.21.15.28:0/1428087068 wait complete. 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.147+0000 7f174beb9640 1 Processor -- start 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 -- start start 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102650 0x7f174419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744110200 0x7f17441a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1744115ca0 con 0x7f1744102fa0 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1744115b20 con 0x7f1744110200 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174beb9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1744115e20 con 0x7f1744102650 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46912/0 (socket says 172.21.15.28:46912) 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 -- 172.21.15.28:0/1727909364 learned_addr learned my addr 172.21.15.28:0/1727909364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749c55640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102650 0x7f174419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f174a456640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744110200 0x7f17441a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 -- 172.21.15.28:0/1727909364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102650 msgr2=0x7f174419c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102650 0x7f174419c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 -- 172.21.15.28:0/1727909364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744110200 msgr2=0x7f17441a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.148+0000 7f1749454640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744110200 0x7f17441a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f1749454640 1 -- 172.21.15.28:0/1727909364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17441a2580 con 0x7f1744102fa0 2024-09-21T05:43:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f1749454640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f174000e9f0 tx=0x7f174000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174000cd80 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17400108e0 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17441a2870 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740018980 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.149+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17441a8fd0 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.150+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174406c1f0 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.153+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f1740010430 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.154+0000 7f1736ffd640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 0x7f17300b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.154+0000 7f1749c55640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 0x7f17300b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.154+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f17400e76d0 con 0x7f1744102fa0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.154+0000 7f1749c55640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 0x7f17300b8af0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1738009bd0 tx=0x7f1738005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.156+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f17400b2710 con 0x7f1744102fa0 2024-09-21T05:43:54.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.346+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f174419d740 con 0x7f1744102fa0 2024-09-21T05:43:54.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.347+0000 7f1736ffd640 1 -- 172.21.15.28:0/1727909364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+18784 (secure 0 0 0) 0x7f17400b7650 con 0x7f1744102fa0 2024-09-21T05:43:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:43:54.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":66,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:43:53.147929+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 msgr2=0x7f17300b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 0x7f17300b8af0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1738009bd0 tx=0x7f1738005f90 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 msgr2=0x7f174419cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f174000e9f0 tx=0x7f174000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 shutdown_connections 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1744110200 0x7f17441a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17300b65e0 0x7f17300b8af0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1744102fa0 0x7f174419cbb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 --2- 172.21.15.28:0/1727909364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1744102650 0x7f174419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 >> 172.21.15.28:0/1727909364 conn(0x7f1744079760 msgr2=0x7f1744109d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 shutdown_connections 2024-09-21T05:43:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:54.349+0000 7f174beb9640 1 -- 172.21.15.28:0/1727909364 wait complete. 2024-09-21T05:43:54.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:54 smithi181 ceph-mon[34819]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:54 smithi181 ceph-mon[34819]: osdmap e66: 12 total, 12 up, 12 in 2024-09-21T05:43:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:54 smithi028 ceph-mon[31227]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:54 smithi028 ceph-mon[31227]: osdmap e66: 12 total, 12 up, 12 in 2024-09-21T05:43:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:54 smithi129 ceph-mon[34885]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-09-21T05:43:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:54 smithi129 ceph-mon[34885]: osdmap e66: 12 total, 12 up, 12 in 2024-09-21T05:43:54.917 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T05:40:11.086256+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T05:43:54.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T05:43:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:55.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:55 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1727909364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:43:55.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:55 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1727909364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:43:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:55 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1727909364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 -- 172.21.15.28:0/3371872990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f810de70 msgr2=0x7f59f8110380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 --2- 172.21.15.28:0/3371872990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f810de70 0x7f59f8110380 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009ca0 tx=0x7f59f002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 -- 172.21.15.28:0/3371872990 shutdown_connections 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 --2- 172.21.15.28:0/3371872990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f810de70 0x7f59f8110380 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 --2- 172.21.15.28:0/3371872990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81052f0 0x7f59f810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 --2- 172.21.15.28:0/3371872990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f81048c0 0x7f59f8104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 -- 172.21.15.28:0/3371872990 >> 172.21.15.28:0/3371872990 conn(0x7f59f8079770 msgr2=0x7f59f8079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 -- 172.21.15.28:0/3371872990 shutdown_connections 2024-09-21T05:43:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.622+0000 7f59fc932640 1 -- 172.21.15.28:0/3371872990 wait complete. 2024-09-21T05:43:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 Processor -- start 2024-09-21T05:43:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 -- start start 2024-09-21T05:43:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81048c0 0x7f59f819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f810de70 0x7f59f819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8113af0 con 0x7f59f81052f0 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8113970 con 0x7f59f81048c0 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59fc932640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8113c70 con 0x7f59f810de70 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59f659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81048c0 0x7f59f819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47518/0 (socket says 172.21.15.28:47518) 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.623+0000 7f59f5d9b640 1 -- 172.21.15.28:0/70593315 learned_addr learned my addr 172.21.15.28:0/70593315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f6d9d640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f810de70 0x7f59f819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 -- 172.21.15.28:0/70593315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f810de70 msgr2=0x7f59f819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f810de70 0x7f59f819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 -- 172.21.15.28:0/70593315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81048c0 msgr2=0x7f59f819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81048c0 0x7f59f819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 -- 172.21.15.28:0/70593315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f81a0430 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59f5d9b640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00d950 tx=0x7f59ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec017070 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ec009c30 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0047a0 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f81a0720 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.624+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f81026b0 con 0x7f59f81052f0 2024-09-21T05:43:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.625+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f806c200 con 0x7f59f81052f0 2024-09-21T05:43:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.628+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f59ec004020 con 0x7f59f81052f0 2024-09-21T05:43:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.628+0000 7f59e77fe640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 0x7f59e00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.629+0000 7f59f659c640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 0x7f59e00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.629+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f59ec0e6fe0 con 0x7f59f81052f0 2024-09-21T05:43:55.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.629+0000 7f59f659c640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 0x7f59e00b8af0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59e800bbe0 tx=0x7f59e800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:55.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.632+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f59ec020020 con 0x7f59f81052f0 2024-09-21T05:43:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.818+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59f819b5f0 con 0x7f59f81052f0 2024-09-21T05:43:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.819+0000 7f59e77fe640 1 -- 172.21.15.28:0/70593315 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) ==== 93+0+10 (secure 0 0 0) 0x7f59f819b5f0 con 0x7f59f81052f0 2024-09-21T05:43:55.821 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 msgr2=0x7f59e00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 0x7f59e00b8af0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59e800bbe0 tx=0x7f59e800b390 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 msgr2=0x7f59f819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00d950 tx=0x7f59ec00de10 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 shutdown_connections 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59f810de70 0x7f59f819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59e00b65e0 0x7f59e00b8af0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59f81052f0 0x7f59f819aa60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 --2- 172.21.15.28:0/70593315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59f81048c0 0x7f59f819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 >> 172.21.15.28:0/70593315 conn(0x7f59f8079770 msgr2=0x7f59f8107a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 shutdown_connections 2024-09-21T05:43:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:55.821+0000 7f59fc932640 1 -- 172.21.15.28:0/70593315 wait complete. 2024-09-21T05:43:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:56 smithi181 ceph-mon[34819]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:56 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/70593315' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:43:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:56 smithi028 ceph-mon[31227]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:56 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/70593315' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:43:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:56 smithi129 ceph-mon[34885]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:56 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/70593315' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.b on smithi129 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.e on smithi129 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.h on smithi129 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.k on smithi129 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.c on smithi181 2024-09-21T05:43:56.618 INFO:tasks.cephadm:Adding mds.f on smithi181 2024-09-21T05:43:56.619 INFO:tasks.cephadm:Adding mds.i on smithi181 2024-09-21T05:43:56.619 INFO:tasks.cephadm:Adding mds.l on smithi181 2024-09-21T05:43:56.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l' 2024-09-21T05:43:56.881 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 -- 172.21.15.181:0/3363393464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3501094a0 msgr2=0x7fd3501098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 --2- 172.21.15.181:0/3363393464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3501094a0 0x7fd3501098b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd338009900 tx=0x7fd33802a410 comp rx=0 tx=0).stop 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 -- 172.21.15.181:0/3363393464 shutdown_connections 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 --2- 172.21.15.181:0/3363393464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350109df0 0x7fd350110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 --2- 172.21.15.181:0/3363393464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3501094a0 0x7fd3501098b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 --2- 172.21.15.181:0/3363393464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350103b60 0x7fd350103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 -- 172.21.15.181:0/3363393464 >> 172.21.15.181:0/3363393464 conn(0x7fd35007b850 msgr2=0x7fd35007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 -- 172.21.15.181:0/3363393464 shutdown_connections 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 -- 172.21.15.181:0/3363393464 wait complete. 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.339+0000 7fd355455640 1 Processor -- start 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 -- start start 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3501094a0 0x7fd35019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 0x7fd35019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113ab0 con 0x7fd3501094a0 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113930 con 0x7fd350109df0 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd355455640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113c30 con 0x7fd350103b60 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:57.344 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51526/0 (socket says 172.21.15.181:51526) 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 -- 172.21.15.181:0/3271000380 learned_addr learned my addr 172.21.15.181:0/3271000380 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34f7fe640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 0x7fd35019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 -- 172.21.15.181:0/3271000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 msgr2=0x7fd35019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 0x7fd35019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 -- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3501094a0 msgr2=0x7fd35019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd3467fc640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3501094a0 0x7fd35019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3501094a0 0x7fd35019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 -- 172.21.15.181:0/3271000380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3501a0370 con 0x7fd350103b60 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34f7fe640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 0x7fd35019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.340+0000 7fd34effd640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd34000d950 tx=0x7fd34000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.341+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340017070 con 0x7fd350103b60 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.341+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501a0660 con 0x7fd350103b60 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.341+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd340009c30 con 0x7fd350103b60 2024-09-21T05:43:57.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.341+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34000f620 con 0x7fd350103b60 2024-09-21T05:43:57.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.341+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501a6e80 con 0x7fd350103b60 2024-09-21T05:43:57.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.342+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350103f70 con 0x7fd350103b60 2024-09-21T05:43:57.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.344+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fd340010460 con 0x7fd350103b60 2024-09-21T05:43:57.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.345+0000 7fd34cff9640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 0x7fd33c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:57.347 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.345+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7fd3400e6bc0 con 0x7fd350103b60 2024-09-21T05:43:57.347 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.345+0000 7fd3467fc640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 0x7fd33c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:57.347 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.346+0000 7fd3467fc640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 0x7fd33c0b8b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd338006010 tx=0x7fd338000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:57.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.348+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd3400ad8a0 con 0x7fd350103b60 2024-09-21T05:43:57.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.537+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}) -- 0x7fd350063800 con 0x7fd33c0b6610 2024-09-21T05:43:57.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.546+0000 7fd34cff9640 1 -- 172.21.15.181:0/3271000380 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd350063800 con 0x7fd33c0b6610 2024-09-21T05:43:57.548 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mds.all update... 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 msgr2=0x7fd33c0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 0x7fd33c0b8b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd338006010 tx=0x7fd338000bc0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 msgr2=0x7fd35019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd34000d950 tx=0x7fd34000de10 comp rx=0 tx=0).stop 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 shutdown_connections 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd350109df0 0x7fd35019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd33c0b6610 0x7fd33c0b8b20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3501094a0 0x7fd35019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 --2- 172.21.15.181:0/3271000380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd350103b60 0x7fd35019a4c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:57.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 >> 172.21.15.181:0/3271000380 conn(0x7fd35007b850 msgr2=0x7fd350105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:57.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 shutdown_connections 2024-09-21T05:43:57.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T05:43:57.549+0000 7fd355455640 1 -- 172.21.15.181:0/3271000380 wait complete. 2024-09-21T05:43:58.161 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.a.service 2024-09-21T05:43:58.164 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.d.service 2024-09-21T05:43:58.207 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.g.service 2024-09-21T05:43:58.250 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.j.service 2024-09-21T05:43:58.294 DEBUG:teuthology.orchestra.run.smithi129:mds.b> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.b.service 2024-09-21T05:43:58.296 DEBUG:teuthology.orchestra.run.smithi129:mds.e> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.e.service 2024-09-21T05:43:58.340 DEBUG:teuthology.orchestra.run.smithi129:mds.h> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.h.service 2024-09-21T05:43:58.383 DEBUG:teuthology.orchestra.run.smithi129:mds.k> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.k.service 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:43:58.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:58 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.426 DEBUG:teuthology.orchestra.run.smithi181:mds.c> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.c.service 2024-09-21T05:43:58.428 DEBUG:teuthology.orchestra.run.smithi181:mds.f> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.f.service 2024-09-21T05:43:58.444 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:43:58.445 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:58 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.471 DEBUG:teuthology.orchestra.run.smithi181:mds.i> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.i.service 2024-09-21T05:43:58.513 DEBUG:teuthology.orchestra.run.smithi181:mds.l> sudo journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.l.service 2024-09-21T05:43:58.557 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-21T05:43:58.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:43:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:43:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:43:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:43:58.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:43:58.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:43:58.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:43:58.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:58 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:43:58.815 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:43:59.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:59 smithi028 ceph-mon[31227]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:59.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:59 smithi028 ceph-mon[31227]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-09-21T05:43:59.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:43:59 smithi028 ceph-mon[31227]: Deploying daemon mds.l on smithi181 2024-09-21T05:43:59.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:59 smithi181 ceph-mon[34819]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:59.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:59 smithi181 ceph-mon[34819]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-09-21T05:43:59.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:43:59 smithi181 ceph-mon[34819]: Deploying daemon mds.l on smithi181 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 -- 172.21.15.28:0/4071335740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 msgr2=0x7fdb44112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/4071335740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb44112620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009ca0 tx=0x7fdb3c02a950 comp rx=0 tx=0).stop 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 -- 172.21.15.28:0/4071335740 shutdown_connections 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/4071335740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb44112620 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/4071335740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/4071335740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 0x7fdb44102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.401+0000 7fdb4baa5640 1 -- 172.21.15.28:0/4071335740 >> 172.21.15.28:0/4071335740 conn(0x7fdb44079760 msgr2=0x7fdb44079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.402+0000 7fdb4baa5640 1 -- 172.21.15.28:0/4071335740 shutdown_connections 2024-09-21T05:43:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.402+0000 7fdb4baa5640 1 -- 172.21.15.28:0/4071335740 wait complete. 2024-09-21T05:43:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.402+0000 7fdb4baa5640 1 Processor -- start 2024-09-21T05:43:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.402+0000 7fdb4baa5640 1 -- start start 2024-09-21T05:43:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 0x7fdb4419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb441a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44115ca0 con 0x7fdb44110200 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44115b20 con 0x7fdb44102650 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4baa5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdb44115e20 con 0x7fdb44102fa0 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4a042640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb441a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4a042640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb441a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47538/0 (socket says 172.21.15.28:47538) 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb4a042640 1 -- 172.21.15.28:0/2786610815 learned_addr learned my addr 172.21.15.28:0/2786610815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49841640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 0x7fdb4419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 -- 172.21.15.28:0/2786610815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 msgr2=0x7fdb4419c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 0x7fdb4419c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 -- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 msgr2=0x7fdb441a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb441a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 -- 172.21.15.28:0/2786610815 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb441a24a0 con 0x7fdb44102fa0 2024-09-21T05:43:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.403+0000 7fdb49040640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4419cb90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400e9f0 tx=0x7fdb3400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.404+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400ccc0 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.404+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb441a2790 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.404+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb34005020 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.404+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400ce60 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.404+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb441a8fb0 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.405+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb4406c1f0 con 0x7fdb44102fa0 2024-09-21T05:43:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.406+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fdb340051d0 con 0x7fdb44102fa0 2024-09-21T05:43:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.407+0000 7fdb3affd640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 0x7fdb300b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:43:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.407+0000 7fdb49841640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 0x7fdb300b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:43:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.407+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7fdb340e6ff0 con 0x7fdb44102fa0 2024-09-21T05:43:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.408+0000 7fdb49841640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 0x7fdb300b8aa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000abe0 tx=0x7fdb4000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:43:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.410+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdb340adca0 con 0x7fdb44102fa0 2024-09-21T05:43:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:59 smithi129 ceph-mon[34885]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:43:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:59 smithi129 ceph-mon[34885]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-09-21T05:43:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:43:59 smithi129 ceph-mon[34885]: Deploying daemon mds.l on smithi181 2024-09-21T05:43:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.635+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb44102a60 con 0x7fdb44102fa0 2024-09-21T05:43:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.636+0000 7fdb3affd640 1 -- 172.21.15.28:0/2786610815 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fdb44102a60 con 0x7fdb44102fa0 2024-09-21T05:43:59.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:43:59.638 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"btime":"2024-09-21T05:36:12:178942+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-21T05:43:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-09-21T05:43:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 msgr2=0x7fdb300b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 0x7fdb300b8aa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000abe0 tx=0x7fdb4000a2d0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 msgr2=0x7fdb4419cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4419cb90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400e9f0 tx=0x7fdb3400edf0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 shutdown_connections 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb300b6590 0x7fdb300b8aa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44110200 0x7fdb441a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb44102fa0 0x7fdb4419cb90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 --2- 172.21.15.28:0/2786610815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb44102650 0x7fdb4419c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 >> 172.21.15.28:0/2786610815 conn(0x7fdb44079760 msgr2=0x7fdb44109d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 shutdown_connections 2024-09-21T05:43:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:43:59.638+0000 7fdb4baa5640 1 -- 172.21.15.28:0/2786610815 wait complete. 2024-09-21T05:43:59.718 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:43:59 smithi181 systemd[1]: Starting Ceph mds.l for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:43:59.969 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:43:59 smithi181 podman[63718]: 2024-09-21 05:43:59.619085601 +0000 UTC m=+0.015911610 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:43:59.969 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:43:59 smithi181 podman[63718]: 2024-09-21 05:43:59.75723764 +0000 UTC m=+0.154063687 container create 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T05:43:59.969 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:43:59 smithi181 podman[63718]: 2024-09-21 05:43:59.967979688 +0000 UTC m=+0.364805689 container init 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:44:00.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2786610815' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6832/1938433457,v1:172.21.15.181:6833/1938433457] up:boot 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: fsmap 1 up:standby 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:00.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:00 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:43:59 smithi181 podman[63718]: 2024-09-21 05:43:59.970806251 +0000 UTC m=+0.367632262 container start 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mds[63740]: -- 172.21.15.181:0/3627670491 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55acef3bd860 con 0x55acf3565600 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: starting mds.l at 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:44:00 smithi181 bash[63718]: 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:44:00 smithi181 systemd[1]: Started Ceph mds.l for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:00.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2786610815' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6832/1938433457,v1:172.21.15.181:6833/1938433457] up:boot 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: fsmap 1 up:standby 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:00.295 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:00 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:00.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:00.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2786610815' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:00.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6832/1938433457,v1:172.21.15.181:6833/1938433457] up:boot 2024-09-21T05:44:00.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: fsmap 1 up:standby 2024-09-21T05:44:00.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:00.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:00 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:01.441 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:01 smithi129 ceph-mon[34885]: Deploying daemon mds.k on smithi129 2024-09-21T05:44:01.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:01 smithi181 ceph-mon[34819]: Deploying daemon mds.k on smithi129 2024-09-21T05:44:01.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:01 smithi028 ceph-mon[31227]: Deploying daemon mds.k on smithi129 2024-09-21T05:44:02.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:02.531 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:02.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:02 smithi181 ceph-mon[34819]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:02.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:02 smithi028 ceph-mon[31227]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:02 smithi129 ceph-mon[34885]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:02.614 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 05:44:02 smithi129 systemd[1]: Started Ceph mds.k for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 -- 172.21.15.28:0/62857335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c109df0 msgr2=0x7fe20c110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 --2- 172.21.15.28:0/62857335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c109df0 0x7fe20c110620 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe204009ca0 tx=0x7fe20402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 -- 172.21.15.28:0/62857335 shutdown_connections 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 --2- 172.21.15.28:0/62857335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c109df0 0x7fe20c110620 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 --2- 172.21.15.28:0/62857335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c1094a0 0x7fe20c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 --2- 172.21.15.28:0/62857335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c103b60 0x7fe20c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 -- 172.21.15.28:0/62857335 >> 172.21.15.28:0/62857335 conn(0x7fe20c07b850 msgr2=0x7fe20c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 -- 172.21.15.28:0/62857335 shutdown_connections 2024-09-21T05:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.102+0000 7fe211509640 1 -- 172.21.15.28:0/62857335 wait complete. 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 Processor -- start 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 -- start start 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c1094a0 0x7fe20c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 0x7fe20c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113ab0 con 0x7fe20c103b60 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113930 con 0x7fe20c109df0 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.103+0000 7fe211509640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113c30 con 0x7fe20c1094a0 2024-09-21T05:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47560/0 (socket says 172.21.15.28:47560) 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 -- 172.21.15.28:0/2859082870 learned_addr learned my addr 172.21.15.28:0/2859082870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20b7fe640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 0x7fe20c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20a7fc640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c1094a0 0x7fe20c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 -- 172.21.15.28:0/2859082870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c1094a0 msgr2=0x7fe20c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c1094a0 0x7fe20c19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 -- 172.21.15.28:0/2859082870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 msgr2=0x7fe20c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 0x7fe20c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 -- 172.21.15.28:0/2859082870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c1a0390 con 0x7fe20c103b60 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20b7fe640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 0x7fe20c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe20affd640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00e9f0 tx=0x7fe1fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00cd80 con 0x7fe20c103b60 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20c1a0680 con 0x7fe20c103b60 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.104+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc0054d0 con 0x7fe20c103b60 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.105+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc010690 con 0x7fe20c103b60 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.105+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20c1a6de0 con 0x7fe20c103b60 2024-09-21T05:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.106+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe20c103f70 con 0x7fe20c103b60 2024-09-21T05:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.107+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fe1fc004e30 con 0x7fe20c103b60 2024-09-21T05:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.108+0000 7fe1f3fff640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 0x7fe1f40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.109+0000 7fe20a7fc640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 0x7fe1f40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.109+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7fe1fc0e7550 con 0x7fe20c103b60 2024-09-21T05:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.109+0000 7fe20a7fc640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 0x7fe1f40b8a70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe20c109190 tx=0x7fe200005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.112+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe1fc0ae1b0 con 0x7fe20c103b60 2024-09-21T05:44:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.356+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe20c19b550 con 0x7fe20c103b60 2024-09-21T05:44:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.357+0000 7fe1f3fff640 1 -- 172.21.15.28:0/2859082870 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2266 (secure 0 0 0) 0x7fe1fc0a9030 con 0x7fe20c103b60 2024-09-21T05:44:03.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:03.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"btime":"2024-09-21T05:44:03:183511+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T05:44:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-09-21T05:44:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 msgr2=0x7fe1f40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 0x7fe1f40b8a70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe20c109190 tx=0x7fe200005f90 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 msgr2=0x7fe20c19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00e9f0 tx=0x7fe1fc00edf0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 shutdown_connections 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20c109df0 0x7fe20c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe20c1094a0 0x7fe20c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1f40b6560 0x7fe1f40b8a70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 --2- 172.21.15.28:0/2859082870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe20c103b60 0x7fe20c19a420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.359+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 >> 172.21.15.28:0/2859082870 conn(0x7fe20c07b850 msgr2=0x7fe20c1074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.360+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 shutdown_connections 2024-09-21T05:44:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:03.360+0000 7fe211509640 1 -- 172.21.15.28:0/2859082870 wait complete. 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: Deploying daemon mds.j on smithi028 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6832/2193748712,v1:172.21.15.129:6833/2193748712] up:boot 2024-09-21T05:44:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: fsmap 2 up:standby 2024-09-21T05:44:03.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:44:03.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:03 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2859082870' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: Deploying daemon mds.j on smithi028 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6832/2193748712,v1:172.21.15.129:6833/2193748712] up:boot 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: fsmap 2 up:standby 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:44:03.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:03 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2859082870' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: Deploying daemon mds.j on smithi028 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6832/2193748712,v1:172.21.15.129:6833/2193748712] up:boot 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: fsmap 2 up:standby 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:44:03.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:03 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2859082870' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:04.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:04 smithi181 ceph-mon[34819]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:04.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:04 smithi028 ceph-mon[31227]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:04 smithi129 ceph-mon[34885]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:05.811 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:05 smithi028 systemd[1]: Starting Ceph mds.j for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:06.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:06.202 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:05 smithi028 podman[72898]: 2024-09-21 05:44:05.840510642 +0000 UTC m=+0.017007203 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:06.202 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:05 smithi028 podman[72898]: 2024-09-21 05:44:05.947558684 +0000 UTC m=+0.124055245 container create 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T05:44:06.202 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 podman[72898]: 2024-09-21 05:44:06.200265491 +0000 UTC m=+0.376762072 container init 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T05:44:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 podman[72898]: 2024-09-21 05:44:06.204443925 +0000 UTC m=+0.380940498 container start 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mds[72935]: -- 172.21.15.28:0/4025314918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586ee729860 con 0x5586f28d1a80 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j[72913]: starting mds.j at 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 bash[72898]: 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 05:44:06 smithi028 systemd[1]: Started Ceph mds.j for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:06.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:06.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:06.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:06 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:06 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 -- 172.21.15.28:0/3990367910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107c50 msgr2=0x7f0d20110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 --2- 172.21.15.28:0/3990367910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107c50 0x7f0d20110670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009ca0 tx=0x7f0d1802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 -- 172.21.15.28:0/3990367910 shutdown_connections 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 --2- 172.21.15.28:0/3990367910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107c50 0x7f0d20110670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 --2- 172.21.15.28:0/3990367910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107300 0x7f0d20107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 --2- 172.21.15.28:0/3990367910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d20102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 -- 172.21.15.28:0/3990367910 >> 172.21.15.28:0/3990367910 conn(0x7f0d20079e40 msgr2=0x7f0d2007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 -- 172.21.15.28:0/3990367910 shutdown_connections 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.838+0000 7f0d27de7640 1 -- 172.21.15.28:0/3990367910 wait complete. 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 Processor -- start 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 -- start start 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d201030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107c50 0x7f0d20103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20113b00 con 0x7f0d20107300 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20113980 con 0x7f0d20107c50 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d27de7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20113c80 con 0x7f0d20102100 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d25382640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d25382640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42440/0 (socket says 172.21.15.28:42440) 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d25382640 1 -- 172.21.15.28:0/1315995 learned_addr learned my addr 172.21.15.28:0/1315995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d25b83640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d201030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.839+0000 7f0d26384640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107c50 0x7f0d20103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 -- 172.21.15.28:0/1315995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 msgr2=0x7f0d201030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d201030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 -- 172.21.15.28:0/1315995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107c50 msgr2=0x7f0d20103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107c50 0x7f0d20103b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 -- 172.21.15.28:0/1315995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d201a8790 con 0x7f0d20107300 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25b83640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d201030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d25382640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c970 tx=0x7f0d1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10007bb0 con 0x7f0d20107300 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d201a8a80 con 0x7f0d20107300 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d10007d50 con 0x7f0d20107300 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10004e10 con 0x7f0d20107300 2024-09-21T05:44:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.840+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d201a8fc0 con 0x7f0d20107300 2024-09-21T05:44:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.841+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d20102510 con 0x7f0d20107300 2024-09-21T05:44:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.843+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f0d10005000 con 0x7f0d20107300 2024-09-21T05:44:06.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.844+0000 7f0d16ffd640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 0x7f0d0c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:06.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.845+0000 7f0d25b83640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 0x7f0d0c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.845+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f0d100e71e0 con 0x7f0d20107300 2024-09-21T05:44:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.845+0000 7f0d25b83640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 0x7f0d0c0b8af0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c005fd0 tx=0x7f0d1c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:06.847+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0d10014030 con 0x7f0d20107300 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:06 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.081+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d201043a0 con 0x7f0d20107300 2024-09-21T05:44:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.082+0000 7f0d16ffd640 1 -- 172.21.15.28:0/1315995 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3091 (secure 0 0 0) 0x7f0d100adda0 con 0x7f0d20107300 2024-09-21T05:44:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:07.084 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"btime":"2024-09-21T05:44:06:438133+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T05:44:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 msgr2=0x7f0d0c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 0x7f0d0c0b8af0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c005fd0 tx=0x7f0d1c005850 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 msgr2=0x7f0d201035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c970 tx=0x7f0d1000ce30 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 shutdown_connections 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d20107c50 0x7f0d20103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d0c0b65e0 0x7f0d0c0b8af0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d20107300 0x7f0d201035e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 --2- 172.21.15.28:0/1315995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d20102100 0x7f0d201030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 >> 172.21.15.28:0/1315995 conn(0x7f0d20079e40 msgr2=0x7f0d20105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 shutdown_connections 2024-09-21T05:44:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:07.084+0000 7f0d27de7640 1 -- 172.21.15.28:0/1315995 wait complete. 2024-09-21T05:44:07.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:07 smithi181 ceph-mon[34819]: Deploying daemon mds.d on smithi028 2024-09-21T05:44:07.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:07 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6834/2759619114,v1:172.21.15.28:6835/2759619114] up:boot 2024-09-21T05:44:07.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:07 smithi181 ceph-mon[34819]: fsmap 3 up:standby 2024-09-21T05:44:07.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:07 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T05:44:07.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:07 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1315995' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:07 smithi028 ceph-mon[31227]: Deploying daemon mds.d on smithi028 2024-09-21T05:44:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:07 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6834/2759619114,v1:172.21.15.28:6835/2759619114] up:boot 2024-09-21T05:44:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:07 smithi028 ceph-mon[31227]: fsmap 3 up:standby 2024-09-21T05:44:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:07 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T05:44:07.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:07 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1315995' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:07 smithi129 ceph-mon[34885]: Deploying daemon mds.d on smithi028 2024-09-21T05:44:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:07 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6834/2759619114,v1:172.21.15.28:6835/2759619114] up:boot 2024-09-21T05:44:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:07 smithi129 ceph-mon[34885]: fsmap 3 up:standby 2024-09-21T05:44:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:07 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T05:44:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:07 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1315995' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:08 smithi028 ceph-mon[31227]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:08 smithi181 ceph-mon[34819]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:08 smithi129 ceph-mon[34885]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:09.024 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:08 smithi028 systemd[1]: Starting Ceph mds.d for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:09.305 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 podman[73284]: 2024-09-21 05:44:09.022916714 +0000 UTC m=+0.113415266 container create 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T05:44:09.305 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 podman[73284]: 2024-09-21 05:44:08.926164632 +0000 UTC m=+0.016663152 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:09.306 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 podman[73284]: 2024-09-21 05:44:09.266864007 +0000 UTC m=+0.357362527 container init 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T05:44:09.306 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 podman[73284]: 2024-09-21 05:44:09.26986882 +0000 UTC m=+0.360367334 container start 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T05:44:09.561 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 ceph-mds[73303]: -- 172.21.15.28:0/743139698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613d9919860 con 0x5613ddac1a80 2024-09-21T05:44:09.561 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d[73299]: starting mds.d at 2024-09-21T05:44:09.561 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 bash[73284]: 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a 2024-09-21T05:44:09.561 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 05:44:09 smithi028 systemd[1]: Started Ceph mds.d for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:10.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:10.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.676 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: Deploying daemon mds.e on smithi129 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6836/783262834,v1:172.21.15.28:6837/783262834] up:boot 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: fsmap 4 up:standby 2024-09-21T05:44:10.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:10 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T05:44:10.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: Deploying daemon mds.e on smithi129 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6836/783262834,v1:172.21.15.28:6837/783262834] up:boot 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: fsmap 4 up:standby 2024-09-21T05:44:10.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:10 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: Deploying daemon mds.e on smithi129 2024-09-21T05:44:10.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6836/783262834,v1:172.21.15.28:6837/783262834] up:boot 2024-09-21T05:44:10.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: fsmap 4 up:standby 2024-09-21T05:44:10.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:10 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T05:44:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 -- 172.21.15.28:0/3140668374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec103b60 msgr2=0x7f7dec103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 --2- 172.21.15.28:0/3140668374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec103b60 0x7f7dec103f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009900 tx=0x7f7ddc02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 -- 172.21.15.28:0/3140668374 shutdown_connections 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 --2- 172.21.15.28:0/3140668374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec109df0 0x7f7dec110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 --2- 172.21.15.28:0/3140668374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec1094a0 0x7f7dec1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 --2- 172.21.15.28:0/3140668374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec103b60 0x7f7dec103f70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 -- 172.21.15.28:0/3140668374 >> 172.21.15.28:0/3140668374 conn(0x7f7dec07b850 msgr2=0x7f7dec07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 -- 172.21.15.28:0/3140668374 shutdown_connections 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.472+0000 7f7df1936640 1 -- 172.21.15.28:0/3140668374 wait complete. 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.473+0000 7f7df1936640 1 Processor -- start 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.473+0000 7f7df1936640 1 -- start start 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.473+0000 7f7df1936640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 0x7f7dec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec1094a0 0x7f7dec075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec113ab0 con 0x7f7dec109df0 2024-09-21T05:44:11.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec113930 con 0x7f7dec103b60 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec113c30 con 0x7f7dec1094a0 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42466/0 (socket says 172.21.15.28:42466) 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 -- 172.21.15.28:0/140760278 learned_addr learned my addr 172.21.15.28:0/140760278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7dea7fc640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec1094a0 0x7f7dec075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deaffd640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 0x7f7dec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 -- 172.21.15.28:0/140760278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec1094a0 msgr2=0x7f7dec075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec1094a0 0x7f7dec075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 -- 172.21.15.28:0/140760278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 msgr2=0x7f7dec075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 0x7f7dec075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 -- 172.21.15.28:0/140760278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec1a8760 con 0x7f7dec109df0 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deaffd640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 0x7f7dec075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7deb7fe640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7de400eaa0 tx=0x7f7de400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de400cdb0 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7de4005020 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dec1a8a50 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.474+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de40116d0 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.475+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dec1a8f40 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.476+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dec103f70 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.477+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f7de40051d0 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.478+0000 7f7df0934640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 0x7f7dd40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.478+0000 7f7deaffd640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 0x7f7dd40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.478+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f7de40e7560 con 0x7f7dec109df0 2024-09-21T05:44:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.479+0000 7f7deaffd640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 0x7f7dd40b8a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc000c00 tx=0x7f7ddc009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.481+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7de4016030 con 0x7f7dec109df0 2024-09-21T05:44:11.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.719+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dec076a10 con 0x7f7dec109df0 2024-09-21T05:44:11.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.719+0000 7f7df0934640 1 -- 172.21.15.28:0/140760278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3913 (secure 0 0 0) 0x7f7dec076a10 con 0x7f7dec109df0 2024-09-21T05:44:11.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:11.721 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"btime":"2024-09-21T05:44:09:451104+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T05:44:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 msgr2=0x7f7dd40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 0x7f7dd40b8a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc000c00 tx=0x7f7ddc009680 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 msgr2=0x7f7dec076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7de400eaa0 tx=0x7f7de400ef60 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 shutdown_connections 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dd40b6560 0x7f7dd40b8a70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec109df0 0x7f7dec076190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dec1094a0 0x7f7dec075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 --2- 172.21.15.28:0/140760278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dec103b60 0x7f7dec075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 >> 172.21.15.28:0/140760278 conn(0x7f7dec07b850 msgr2=0x7f7dec107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 shutdown_connections 2024-09-21T05:44:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:11.722+0000 7f7df1936640 1 -- 172.21.15.28:0/140760278 wait complete. 2024-09-21T05:44:11.864 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 05:44:11 smithi129 systemd[1]: Started Ceph mds.e for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:12 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/140760278' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:12.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:12 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/140760278' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:12.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:12.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:12.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:12.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:12.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:12.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:12 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/140760278' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:13.424 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 systemd[1]: Starting Ceph mds.f for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-mon[34819]: Deploying daemon mds.f on smithi181 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6834/2842004697,v1:172.21.15.129:6835/2842004697] up:boot 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-mon[34819]: fsmap 5 up:standby 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 podman[63990]: 2024-09-21 05:44:13.46546676 +0000 UTC m=+0.015756568 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:13.675 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 podman[63990]: 2024-09-21 05:44:13.575051406 +0000 UTC m=+0.125341246 container create 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:44:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:13 smithi028 ceph-mon[31227]: Deploying daemon mds.f on smithi181 2024-09-21T05:44:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:13 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6834/2842004697,v1:172.21.15.129:6835/2842004697] up:boot 2024-09-21T05:44:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:13 smithi028 ceph-mon[31227]: fsmap 5 up:standby 2024-09-21T05:44:13.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:44:13.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:13 smithi129 ceph-mon[34885]: Deploying daemon mds.f on smithi181 2024-09-21T05:44:13.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:13 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6834/2842004697,v1:172.21.15.129:6835/2842004697] up:boot 2024-09-21T05:44:13.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:13 smithi129 ceph-mon[34885]: fsmap 5 up:standby 2024-09-21T05:44:13.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 podman[63990]: 2024-09-21 05:44:13.769193773 +0000 UTC m=+0.319483570 container init 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 podman[63990]: 2024-09-21 05:44:13.771997173 +0000 UTC m=+0.322286970 container start 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-mds[64010]: -- 172.21.15.181:0/4044175976 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55732d949c20 con 0x557331af1180 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: starting mds.f at 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 bash[63990]: 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d 2024-09-21T05:44:14.044 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:44:13 smithi181 systemd[1]: Started Ceph mds.f for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:14.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:14.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:14.616 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:14.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:14.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 -- 172.21.15.28:0/1473158142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18102ae0 msgr2=0x7f2a18102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 --2- 172.21.15.28:0/1473158142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18102ae0 0x7f2a18102ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009900 tx=0x7f2a1402a410 comp rx=0 tx=0).stop 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 -- 172.21.15.28:0/1473158142 shutdown_connections 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 --2- 172.21.15.28:0/1473158142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a18108d20 0x7f2a1810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 --2- 172.21.15.28:0/1473158142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a181083d0 0x7f2a181087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 --2- 172.21.15.28:0/1473158142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18102ae0 0x7f2a18102ef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.217+0000 7f2a1f643640 1 -- 172.21.15.28:0/1473158142 >> 172.21.15.28:0/1473158142 conn(0x7f2a18079760 msgr2=0x7f2a18079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.218+0000 7f2a1f643640 1 -- 172.21.15.28:0/1473158142 shutdown_connections 2024-09-21T05:44:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.218+0000 7f2a1f643640 1 -- 172.21.15.28:0/1473158142 wait complete. 2024-09-21T05:44:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.218+0000 7f2a1f643640 1 Processor -- start 2024-09-21T05:44:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.218+0000 7f2a1f643640 1 -- start start 2024-09-21T05:44:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a18102ae0 0x7f2a1810fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a181083d0 0x7f2a18110320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18113ac0 con 0x7f2a18108d20 2024-09-21T05:44:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18113940 con 0x7f2a18102ae0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1f643640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18113c40 con 0x7f2a181083d0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1cbde640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a181083d0 0x7f2a18110320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34410/0 (socket says 172.21.15.28:34410) 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 -- 172.21.15.28:0/421049986 learned_addr learned my addr 172.21.15.28:0/421049986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1d3df640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a18102ae0 0x7f2a1810fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 -- 172.21.15.28:0/421049986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a181083d0 msgr2=0x7f2a18110320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a181083d0 0x7f2a18110320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 -- 172.21.15.28:0/421049986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a18102ae0 msgr2=0x7f2a1810fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a18102ae0 0x7f2a1810fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 -- 172.21.15.28:0/421049986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a181a87d0 con 0x7f2a18108d20 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a1dbe0640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000cad0 tx=0x7f2a1000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.219+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000ae20 con 0x7f2a18108d20 2024-09-21T05:44:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.220+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a10005020 con 0x7f2a18108d20 2024-09-21T05:44:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.220+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a181a8ac0 con 0x7f2a18108d20 2024-09-21T05:44:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.220+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000f6b0 con 0x7f2a18108d20 2024-09-21T05:44:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.220+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a181a9000 con 0x7f2a18108d20 2024-09-21T05:44:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.221+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a18103d10 con 0x7f2a18108d20 2024-09-21T05:44:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.222+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f2a100051d0 con 0x7f2a18108d20 2024-09-21T05:44:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.223+0000 7f2a0a7fc640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 0x7f2a040b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.223+0000 7f2a1d3df640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 0x7f2a040b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.223+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f2a100e7250 con 0x7f2a18108d20 2024-09-21T05:44:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.224+0000 7f2a1d3df640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 0x7f2a040b8a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2a14000c00 tx=0x7f2a14031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.226+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2a10014030 con 0x7f2a18108d20 2024-09-21T05:44:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.455+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a18102ef0 con 0x7f2a18108d20 2024-09-21T05:44:15.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:15 smithi028 ceph-mon[31227]: Deploying daemon mds.c on smithi181 2024-09-21T05:44:15.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:15 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:boot 2024-09-21T05:44:15.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:15 smithi028 ceph-mon[31227]: fsmap 6 up:standby 2024-09-21T05:44:15.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:15 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:44:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.455+0000 7f2a0a7fc640 1 -- 172.21.15.28:0/421049986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5569 (secure 0 0 0) 0x7f2a100b2290 con 0x7f2a18108d20 2024-09-21T05:44:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:15.459 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"btime":"2024-09-21T05:44:14:423071+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24638,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24664,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-21T05:44:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 msgr2=0x7f2a040b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 0x7f2a040b8a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2a14000c00 tx=0x7f2a14031400 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 msgr2=0x7f2a18110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000cad0 tx=0x7f2a1000cf90 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 shutdown_connections 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2a040b6560 0x7f2a040b8a70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a18108d20 0x7f2a18110860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a181083d0 0x7f2a18110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 --2- 172.21.15.28:0/421049986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a18102ae0 0x7f2a1810fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 >> 172.21.15.28:0/421049986 conn(0x7f2a18079760 msgr2=0x7f2a181015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 shutdown_connections 2024-09-21T05:44:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:15.458+0000 7f2a1f643640 1 -- 172.21.15.28:0/421049986 wait complete. 2024-09-21T05:44:15.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:15 smithi181 ceph-mon[34819]: Deploying daemon mds.c on smithi181 2024-09-21T05:44:15.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:15 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:boot 2024-09-21T05:44:15.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:15 smithi181 ceph-mon[34819]: fsmap 6 up:standby 2024-09-21T05:44:15.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:15 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:44:15.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:15 smithi129 ceph-mon[34885]: Deploying daemon mds.c on smithi181 2024-09-21T05:44:15.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:15 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:boot 2024-09-21T05:44:15.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:15 smithi129 ceph-mon[34885]: fsmap 6 up:standby 2024-09-21T05:44:15.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:15 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:44:15.929 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:15 smithi181 systemd[1]: Starting Ceph mds.c for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:16.184 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:15 smithi181 podman[64258]: 2024-09-21 05:44:15.927870627 +0000 UTC m=+0.105105465 container create 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T05:44:16.184 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:15 smithi181 podman[64258]: 2024-09-21 05:44:15.838454457 +0000 UTC m=+0.015689250 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:16.185 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 podman[64258]: 2024-09-21 05:44:16.146971529 +0000 UTC m=+0.324206323 container init 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T05:44:16.185 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 podman[64258]: 2024-09-21 05:44:16.149889177 +0000 UTC m=+0.327123971 container start 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T05:44:16.440 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mds[64277]: -- 172.21.15.181:0/2437468217 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc60217860 con 0x55bc643bf600 2024-09-21T05:44:16.440 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: starting mds.c at 2024-09-21T05:44:16.440 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 bash[64258]: 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb 2024-09-21T05:44:16.440 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:44:16 smithi181 systemd[1]: Started Ceph mds.c for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/421049986' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:16.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:16 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/421049986' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:16 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/421049986' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:16.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:16 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:17.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:17 smithi129 ceph-mon[34885]: Deploying daemon mds.h on smithi129 2024-09-21T05:44:17.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:17 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:boot 2024-09-21T05:44:17.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:17 smithi129 ceph-mon[34885]: fsmap 7 up:standby 2024-09-21T05:44:17.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:17 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:44:17.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:17 smithi181 ceph-mon[34819]: Deploying daemon mds.h on smithi129 2024-09-21T05:44:17.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:17 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:boot 2024-09-21T05:44:17.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:17 smithi181 ceph-mon[34819]: fsmap 7 up:standby 2024-09-21T05:44:17.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:17 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:44:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:17 smithi028 ceph-mon[31227]: Deploying daemon mds.h on smithi129 2024-09-21T05:44:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:17 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:boot 2024-09-21T05:44:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:17 smithi028 ceph-mon[31227]: fsmap 7 up:standby 2024-09-21T05:44:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:17 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:44:18.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:18 smithi028 ceph-mon[31227]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:18 smithi181 ceph-mon[34819]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:18 smithi129 ceph-mon[34885]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:18.864 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 05:44:18 smithi129 systemd[1]: Started Ceph mds.h for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/3926467681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc4105380 msgr2=0x7f3bc410d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/3926467681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc4105380 0x7f3bc410d8d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009900 tx=0x7f3bb802a410 comp rx=0 tx=0).stop 2024-09-21T05:44:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/3926467681 shutdown_connections 2024-09-21T05:44:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/3926467681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc410de60 0x7f3bc4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/3926467681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc4105380 0x7f3bc410d8d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/3926467681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc4104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/3926467681 >> 172.21.15.28:0/3926467681 conn(0x7f3bc40797b0 msgr2=0x7f3bc4079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/3926467681 shutdown_connections 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.961+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/3926467681 wait complete. 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 Processor -- start 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 -- start start 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc4105380 0x7f3bc419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113aa0 con 0x7f3bc410de60 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113920 con 0x7f3bc4104a30 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc8e6a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113c20 con 0x7f3bc4105380 2024-09-21T05:44:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34430/0 (socket says 172.21.15.28:34430) 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc2d9d640 1 -- 172.21.15.28:0/1838911965 learned_addr learned my addr 172.21.15.28:0/1838911965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.962+0000 7f3bc1d9b640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc4105380 0x7f3bc419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc259c640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 -- 172.21.15.28:0/1838911965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc4105380 msgr2=0x7f3bc419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc4105380 0x7f3bc419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 -- 172.21.15.28:0/1838911965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 msgr2=0x7f3bc419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 -- 172.21.15.28:0/1838911965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc41a0420 con 0x7f3bc410de60 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc259c640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc2d9d640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00eaa0 tx=0x7f3bbc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00cdb0 con 0x7f3bc410de60 2024-09-21T05:44:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc41a0710 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bbc005020 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0116d0 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.963+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc41a6e70 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.964+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc4104e40 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.966+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f3bbc0051d0 con 0x7f3bc410de60 2024-09-21T05:44:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.967+0000 7f3bb37fe640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 0x7f3bac0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.967+0000 7f3bc259c640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 0x7f3bac0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.967+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f3bbc0e7530 con 0x7f3bc410de60 2024-09-21T05:44:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.968+0000 7f3bc259c640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 0x7f3bac0b8a70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40058c0 tx=0x7f3bb4005ec0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:18.974+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3bbc016030 con 0x7f3bc410de60 2024-09-21T05:44:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.205+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc419b5e0 con 0x7f3bc410de60 2024-09-21T05:44:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.206+0000 7f3bb37fe640 1 -- 172.21.15.28:0/1838911965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6397 (secure 0 0 0) 0x7f3bc419b5e0 con 0x7f3bc410de60 2024-09-21T05:44:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:19.209 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"btime":"2024-09-21T05:44:16:437084+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24638,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24664,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-21T05:44:19.209 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-21T05:44:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 msgr2=0x7f3bac0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 0x7f3bac0b8a70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40058c0 tx=0x7f3bb4005ec0 comp rx=0 tx=0).stop 2024-09-21T05:44:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 msgr2=0x7f3bc419fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00eaa0 tx=0x7f3bbc00ef60 comp rx=0 tx=0).stop 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 shutdown_connections 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bac0b6560 0x7f3bac0b8a70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc410de60 0x7f3bc419fcf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc4105380 0x7f3bc419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 --2- 172.21.15.28:0/1838911965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc4104a30 0x7f3bc419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 >> 172.21.15.28:0/1838911965 conn(0x7f3bc40797b0 msgr2=0x7f3bc41008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 shutdown_connections 2024-09-21T05:44:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:19.208+0000 7f3bc8e6a640 1 -- 172.21.15.28:0/1838911965 wait complete. 2024-09-21T05:44:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: Deploying daemon mds.i on smithi181 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1838911965' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:boot 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: fsmap 8 up:standby 2024-09-21T05:44:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:19 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: Deploying daemon mds.i on smithi181 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1838911965' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:20.038 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:boot 2024-09-21T05:44:20.039 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: fsmap 8 up:standby 2024-09-21T05:44:20.039 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:19 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:44:20.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:20.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: Deploying daemon mds.i on smithi181 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1838911965' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:boot 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: fsmap 8 up:standby 2024-09-21T05:44:20.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:19 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:44:20.761 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:20 smithi181 systemd[1]: Starting Ceph mds.i for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:21.036 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:20 smithi181 podman[64529]: 2024-09-21 05:44:20.662528556 +0000 UTC m=+0.016010036 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:21.036 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:20 smithi181 podman[64529]: 2024-09-21 05:44:20.763361822 +0000 UTC m=+0.116843292 container create 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T05:44:21.036 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:20 smithi181 podman[64529]: 2024-09-21 05:44:20.999175743 +0000 UTC m=+0.352657202 container init 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T05:44:21.037 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:21 smithi181 podman[64529]: 2024-09-21 05:44:21.001984904 +0000 UTC m=+0.355466374 container start 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T05:44:21.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:21 smithi181 ceph-mds[64549]: -- 172.21.15.181:0/3553156623 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56149ad83860 con 0x56149ef2b600 2024-09-21T05:44:21.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: starting mds.i at 2024-09-21T05:44:21.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:21 smithi181 bash[64529]: 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 2024-09-21T05:44:21.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:21 smithi181 systemd[1]: Started Ceph mds.i for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:21.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:22.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:22.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: Deploying daemon mds.g on smithi028 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:boot 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: fsmap 9 up:standby 2024-09-21T05:44:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:22 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: Deploying daemon mds.g on smithi028 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:boot 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: fsmap 9 up:standby 2024-09-21T05:44:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:22 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:22.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: Deploying daemon mds.g on smithi028 2024-09-21T05:44:22.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:boot 2024-09-21T05:44:22.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: fsmap 9 up:standby 2024-09-21T05:44:22.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:22 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:44:22.676 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 -- 172.21.15.28:0/2476621884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 msgr2=0x7f50ec10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 --2- 172.21.15.28:0/2476621884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec10f520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009900 tx=0x7f50e402a410 comp rx=0 tx=0).stop 2024-09-21T05:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 -- 172.21.15.28:0/2476621884 shutdown_connections 2024-09-21T05:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 --2- 172.21.15.28:0/2476621884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec10f520 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 --2- 172.21.15.28:0/2476621884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec1083a0 0x7f50ec1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 --2- 172.21.15.28:0/2476621884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec102a60 0x7f50ec102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 -- 172.21.15.28:0/2476621884 >> 172.21.15.28:0/2476621884 conn(0x7f50ec079730 msgr2=0x7f50ec079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 -- 172.21.15.28:0/2476621884 shutdown_connections 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.269+0000 7f50f3539640 1 -- 172.21.15.28:0/2476621884 wait complete. 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 Processor -- start 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 -- start start 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec102a60 0x7f50ec19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 0x7f50ec19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec113a80 con 0x7f50ec108cf0 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec113900 con 0x7f50ec102a60 2024-09-21T05:44:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.270+0000 7f50f3539640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec113c00 con 0x7f50ec1083a0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34450/0 (socket says 172.21.15.28:34450) 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 -- 172.21.15.28:0/2322413959 learned_addr learned my addr 172.21.15.28:0/2322413959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f12d5640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec102a60 0x7f50ec19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f0ad4640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 0x7f50ec19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 -- 172.21.15.28:0/2322413959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 msgr2=0x7f50ec19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 0x7f50ec19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 -- 172.21.15.28:0/2322413959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec102a60 msgr2=0x7f50ec19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec102a60 0x7f50ec19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 -- 172.21.15.28:0/2322413959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec1a0320 con 0x7f50ec108cf0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f0ad4640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 0x7f50ec19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f1ad6640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009a20 tx=0x7f50e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e403f030 con 0x7f50ec108cf0 2024-09-21T05:44:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50e4003aa0 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec1a05b0 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4035070 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.271+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec1a0a90 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.272+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec102e70 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.274+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f50e4042030 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.275+0000 7f50ea7fc640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 0x7f50d40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.275+0000 7f50f12d5640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 0x7f50d40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.276+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f50e4107af0 con 0x7f50ec108cf0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.276+0000 7f50f12d5640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 0x7f50d40b8aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005c40 tx=0x7f50dc00a780 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.279+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f50e40ce750 con 0x7f50ec108cf0 2024-09-21T05:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.512+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50ec02e640 con 0x7f50ec108cf0 2024-09-21T05:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.512+0000 7f50ea7fc640 1 -- 172.21.15.28:0/2322413959 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8055 (secure 0 0 0) 0x7f50e40b3e20 con 0x7f50ec108cf0 2024-09-21T05:44:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:23.515 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"btime":"2024-09-21T05:44:21:751912+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24638,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24653,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24664,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24688,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T05:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 msgr2=0x7f50d40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 0x7f50d40b8aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005c40 tx=0x7f50dc00a780 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 msgr2=0x7f50ec19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009a20 tx=0x7f50e4004270 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 shutdown_connections 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f50d40b6590 0x7f50d40b8aa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec108cf0 0x7f50ec19fcb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50ec1083a0 0x7f50ec19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 --2- 172.21.15.28:0/2322413959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ec102a60 0x7f50ec19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 >> 172.21.15.28:0/2322413959 conn(0x7f50ec079730 msgr2=0x7f50ec104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 shutdown_connections 2024-09-21T05:44:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:23.515+0000 7f50f3539640 1 -- 172.21.15.28:0/2322413959 wait complete. 2024-09-21T05:44:24.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:24 smithi028 ceph-mon[31227]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:24.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:24 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2322413959' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:24.417 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:24 smithi129 ceph-mon[34885]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:24.417 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:24 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2322413959' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:24.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:24 smithi181 ceph-mon[34819]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:24 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2322413959' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:25.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: Deploying daemon mds.a on smithi028 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6838/2924711217,v1:172.21.15.28:6839/2924711217] up:boot 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: fsmap 10 up:standby 2024-09-21T05:44:25.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:25 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:44:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:25.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: Deploying daemon mds.a on smithi028 2024-09-21T05:44:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6838/2924711217,v1:172.21.15.28:6839/2924711217] up:boot 2024-09-21T05:44:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: fsmap 10 up:standby 2024-09-21T05:44:25.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:25 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: Deploying daemon mds.a on smithi028 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6838/2924711217,v1:172.21.15.28:6839/2924711217] up:boot 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: fsmap 10 up:standby 2024-09-21T05:44:25.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:25 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:44:26.303 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 systemd[1]: Starting Ceph mds.a for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:26.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:26 smithi028 ceph-mon[31227]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:26.561 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 podman[74290]: 2024-09-21 05:44:26.273944086 +0000 UTC m=+0.016615898 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:26.562 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 podman[74290]: 2024-09-21 05:44:26.384303877 +0000 UTC m=+0.126975674 container create 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:44:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:26 smithi129 ceph-mon[34885]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:26 smithi181 ceph-mon[34819]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:26.882 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 podman[74290]: 2024-09-21 05:44:26.595329336 +0000 UTC m=+0.338001141 container init 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:44:26.882 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 podman[74290]: 2024-09-21 05:44:26.598516038 +0000 UTC m=+0.341187843 container start 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T05:44:26.882 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 ceph-mds[74334]: -- 172.21.15.28:0/1124666973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4d6b31860 con 0x55c4dacd9600 2024-09-21T05:44:26.882 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: starting mds.a at 2024-09-21T05:44:26.882 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 bash[74290]: 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 2024-09-21T05:44:26.883 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:44:26 smithi028 systemd[1]: Started Ceph mds.a for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.223+0000 7f9b145ac640 1 -- 172.21.15.28:0/1059293200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c1083e0 msgr2=0x7f9b0c1087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.223+0000 7f9b145ac640 1 --2- 172.21.15.28:0/1059293200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c1087f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009900 tx=0x7f9afc02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 -- 172.21.15.28:0/1059293200 shutdown_connections 2024-09-21T05:44:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 --2- 172.21.15.28:0/1059293200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c108d30 0x7f9b0c10f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 --2- 172.21.15.28:0/1059293200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c1087f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 --2- 172.21.15.28:0/1059293200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 -- 172.21.15.28:0/1059293200 >> 172.21.15.28:0/1059293200 conn(0x7f9b0c079750 msgr2=0x7f9b0c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 -- 172.21.15.28:0/1059293200 shutdown_connections 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 -- 172.21.15.28:0/1059293200 wait complete. 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 Processor -- start 2024-09-21T05:44:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.224+0000 7f9b145ac640 1 -- start start 2024-09-21T05:44:27.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c108d30 0x7f9b0c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1129f0 con 0x7f9b0c102aa0 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c112870 con 0x7f9b0c1083e0 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b145ac640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c112b70 con 0x7f9b0c108d30 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 -- 172.21.15.28:0/373226497 learned_addr learned my addr 172.21.15.28:0/373226497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b11b47640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12b49640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c108d30 0x7f9b0c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 -- 172.21.15.28:0/373226497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c108d30 msgr2=0x7f9b0c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c108d30 0x7f9b0c19fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 -- 172.21.15.28:0/373226497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c1083e0 msgr2=0x7f9b0c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.225+0000 7f9b12348640 1 -- 172.21.15.28:0/373226497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c1a0440 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b12348640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800d950 tx=0x7f9b0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08017030 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b08004020 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0c1a0730 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b080047a0 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.226+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0c1a6e90 con 0x7f9b0c102aa0 2024-09-21T05:44:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.227+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b0c102eb0 con 0x7f9b0c102aa0 2024-09-21T05:44:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.229+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f9b08009a30 con 0x7f9b0c102aa0 2024-09-21T05:44:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.230+0000 7f9b037fe640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 0x7f9af80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.230+0000 7f9b11b47640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 0x7f9af80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.230+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f9b080e6fc0 con 0x7f9b0c102aa0 2024-09-21T05:44:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.230+0000 7f9b11b47640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 0x7f9af80b8af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009230 tx=0x7f9afc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.233+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9b080adca0 con 0x7f9b0c102aa0 2024-09-21T05:44:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.464+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b0c19b600 con 0x7f9b0c102aa0 2024-09-21T05:44:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.465+0000 7f9b037fe640 1 -- 172.21.15.28:0/373226497 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9707 (secure 0 0 0) 0x7f9b0c19b600 con 0x7f9b0c102aa0 2024-09-21T05:44:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:27.467 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"btime":"2024-09-21T05:44:27:302166+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14850,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24638,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24653,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24664,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24688,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T05:44:27.467 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-09-21T05:44:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 msgr2=0x7f9af80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 0x7f9af80b8af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009230 tx=0x7f9afc031400 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 msgr2=0x7f9b0c19a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800d950 tx=0x7f9b0800de10 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 shutdown_connections 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0c108d30 0x7f9b0c19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b0c1083e0 0x7f9b0c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9af80b65e0 0x7f9af80b8af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 --2- 172.21.15.28:0/373226497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b0c102aa0 0x7f9b0c19a4d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 >> 172.21.15.28:0/373226497 conn(0x7f9b0c079750 msgr2=0x7f9b0c101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 shutdown_connections 2024-09-21T05:44:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:27.467+0000 7f9b145ac640 1 -- 172.21.15.28:0/373226497 wait complete. 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: Deploying daemon mds.b on smithi129 2024-09-21T05:44:28.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6840/2765835727,v1:172.21.15.28:6841/2765835727] up:boot 2024-09-21T05:44:28.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: fsmap 11 up:standby 2024-09-21T05:44:28.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:44:28.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/373226497' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:28.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:28.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: Deploying daemon mds.b on smithi129 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6840/2765835727,v1:172.21.15.28:6841/2765835727] up:boot 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: fsmap 11 up:standby 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:44:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/373226497' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:28.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: Deploying daemon mds.b on smithi129 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6840/2765835727,v1:172.21.15.28:6841/2765835727] up:boot 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: fsmap 11 up:standby 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:44:28.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/373226497' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:28.864 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:28 smithi129 systemd[1]: Starting Ceph mds.b for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:44:29.221 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:28 smithi129 podman[63421]: 2024-09-21 05:44:28.847422684 +0000 UTC m=+0.015889907 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:44:29.221 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:28 smithi129 podman[63421]: 2024-09-21 05:44:28.967783242 +0000 UTC m=+0.136250499 container create 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0) 2024-09-21T05:44:29.221 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 podman[63421]: 2024-09-21 05:44:29.220169243 +0000 UTC m=+0.388636458 container init 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T05:44:29.614 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 podman[63421]: 2024-09-21 05:44:29.222933114 +0000 UTC m=+0.391400328 container start 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T05:44:29.614 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 ceph-mds[63440]: -- 172.21.15.129:0/2292593176 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d2ac01860 con 0x561d2eda9a80 2024-09-21T05:44:29.615 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: starting mds.b at 2024-09-21T05:44:29.615 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 bash[63421]: 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 2024-09-21T05:44:29.615 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:44:29 smithi129 systemd[1]: Started Ceph mds.b for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:44:30.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:44:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:boot 2024-09-21T05:44:30.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: fsmap 12 up:standby 2024-09-21T05:44:30.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:30 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:boot 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: fsmap 12 up:standby 2024-09-21T05:44:30.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:30 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:boot 2024-09-21T05:44:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: fsmap 12 up:standby 2024-09-21T05:44:30.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:30 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 -- 172.21.15.28:0/3194643613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 msgr2=0x7f4008103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3194643613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f4008103f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009ca0 tx=0x7f3ff802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 -- 172.21.15.28:0/3194643613 shutdown_connections 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3194643613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4008109df0 0x7f4008110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3194643613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40081094a0 0x7f40081098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3194643613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f4008103f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 -- 172.21.15.28:0/3194643613 >> 172.21.15.28:0/3194643613 conn(0x7f400807b850 msgr2=0x7f400807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 -- 172.21.15.28:0/3194643613 shutdown_connections 2024-09-21T05:44:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.120+0000 7f400fbb0640 1 -- 172.21.15.28:0/3194643613 wait complete. 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 Processor -- start 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 -- start start 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40081094a0 0x7f400819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 0x7f400819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4008113ab0 con 0x7f4008103b60 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4008113930 con 0x7f4008109df0 2024-09-21T05:44:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400fbb0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4008113c30 con 0x7f40081094a0 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53544/0 (socket says 172.21.15.28:53544) 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 -- 172.21.15.28:0/3026845746 learned_addr learned my addr 172.21.15.28:0/3026845746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d14b640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40081094a0 0x7f400819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400e14d640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 0x7f400819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 -- 172.21.15.28:0/3026845746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40081094a0 msgr2=0x7f400819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40081094a0 0x7f400819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 -- 172.21.15.28:0/3026845746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 msgr2=0x7f400819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 0x7f400819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400d94c640 1 -- 172.21.15.28:0/3026845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40081a0330 con 0x7f4008103b60 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.121+0000 7f400e14d640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 0x7f400819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f400d94c640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8006010 tx=0x7f3ff80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:31.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8003bf0 con 0x7f4008103b60 2024-09-21T05:44:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff8003d90 con 0x7f4008103b60 2024-09-21T05:44:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40081a05c0 con 0x7f4008103b60 2024-09-21T05:44:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff802fc20 con 0x7f4008103b60 2024-09-21T05:44:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.122+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40081a6e40 con 0x7f4008103b60 2024-09-21T05:44:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.123+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4008104de0 con 0x7f4008103b60 2024-09-21T05:44:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.125+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f3ff80046a0 con 0x7f4008103b60 2024-09-21T05:44:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.125+0000 7f4006ffd640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 0x7f3ff00b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.125+0000 7f400d14b640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 0x7f3ff00b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.125+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f3ff8108240 con 0x7f4008103b60 2024-09-21T05:44:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.126+0000 7f400d14b640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 0x7f3ff00b8b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4008106450 tx=0x7f3ffc00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.130+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3ff80ceea0 con 0x7f4008103b60 2024-09-21T05:44:31.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.381+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f400810bbc0 con 0x7f4008103b60 2024-09-21T05:44:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.382+0000 7f4006ffd640 1 -- 172.21.15.28:0/3026845746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10536 (secure 0 0 0) 0x7f3ff80d3280 con 0x7f4008103b60 2024-09-21T05:44:31.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:31.384 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"btime":"2024-09-21T05:44:29:738437+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14850,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24614,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24622,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24638,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24653,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24664,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24688,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-21T05:44:31.385 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.384+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 msgr2=0x7f3ff00b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.384+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 0x7f3ff00b8b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4008106450 tx=0x7f3ffc00a2d0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.384+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 msgr2=0x7f400819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.384+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8006010 tx=0x7f3ff80319f0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 shutdown_connections 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4008109df0 0x7f400819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40081094a0 0x7f400819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3ff00b6610 0x7f3ff00b8b20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 --2- 172.21.15.28:0/3026845746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4008103b60 0x7f400819a480 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 >> 172.21.15.28:0/3026845746 conn(0x7f400807b850 msgr2=0x7f4008107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 shutdown_connections 2024-09-21T05:44:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:31.385+0000 7f400fbb0640 1 -- 172.21.15.28:0/3026845746 wait complete. 2024-09-21T05:44:32.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3026845746' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:31 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3026845746' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:32.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:31 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph --format=json-pretty osd lspools 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3026845746' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:31 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:33.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:32 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:33.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:32 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:33.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:32 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:33.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:32 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:33.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:32 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:33.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:32 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:33.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:32 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:33.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:32 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:33.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:32 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:33.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:32 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:33.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:32 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:33.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:32 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 -- 172.21.15.28:0/1691035340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 msgr2=0x7f3ca81087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 --2- 172.21.15.28:0/1691035340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca81087e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009900 tx=0x7f3c9c02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 -- 172.21.15.28:0/1691035340 shutdown_connections 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 --2- 172.21.15.28:0/1691035340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 0x7f3ca810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 --2- 172.21.15.28:0/1691035340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca81087e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 --2- 172.21.15.28:0/1691035340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 -- 172.21.15.28:0/1691035340 >> 172.21.15.28:0/1691035340 conn(0x7f3ca80797b0 msgr2=0x7f3ca8079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 -- 172.21.15.28:0/1691035340 shutdown_connections 2024-09-21T05:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.437+0000 7f3cad27f640 1 -- 172.21.15.28:0/1691035340 wait complete. 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 Processor -- start 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 -- start start 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 0x7f3ca819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca81129e0 con 0x7f3ca81083d0 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca8112860 con 0x7f3ca8108d20 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.438+0000 7f3cad27f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca8112b60 con 0x7f3ca8102ae0 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca819b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53564/0 (socket says 172.21.15.28:53564) 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca67fc640 1 -- 172.21.15.28:0/2889252536 learned_addr learned my addr 172.21.15.28:0/2889252536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca77fe640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 0x7f3ca819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 -- 172.21.15.28:0/2889252536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 msgr2=0x7f3ca819bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 0x7f3ca819bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 -- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 msgr2=0x7f3ca819b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca819b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 -- 172.21.15.28:0/2889252536 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca81991a0 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.439+0000 7f3ca6ffd640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca819b330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800d6d0 tx=0x7f3c9800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.440+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98017070 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.440+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c98009820 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.440+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca8199490 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.440+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98004c40 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.440+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca8199a20 con 0x7f3ca8102ae0 2024-09-21T05:44:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.441+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca8102ef0 con 0x7f3ca8102ae0 2024-09-21T05:44:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.442+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f3c98005220 con 0x7f3ca8102ae0 2024-09-21T05:44:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.444+0000 7f3c8ffff640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 0x7f3c900b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:33.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.444+0000 7f3ca67fc640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 0x7f3c900b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:33.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.444+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f3c980e69b0 con 0x7f3ca8102ae0 2024-09-21T05:44:33.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.444+0000 7f3ca67fc640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 0x7f3c900b8b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8107be0 tx=0x7f3c9c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:33.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.446+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c980b19d0 con 0x7f3ca8102ae0 2024-09-21T05:44:33.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.659+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3ca819af10 con 0x7f3ca8102ae0 2024-09-21T05:44:33.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.660+0000 7f3c8ffff640 1 -- 172.21.15.28:0/2889252536 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) ==== 84+0+65 (secure 0 0 0) 0x7f3c980b6910 con 0x7f3ca8102ae0 2024-09-21T05:44:33.662 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:33.662 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-21T05:44:33.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:44:33.662 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-21T05:44:33.663 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-21T05:44:33.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:44:33.663 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 msgr2=0x7f3c900b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 0x7f3c900b8b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8107be0 tx=0x7f3c9c009560 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 msgr2=0x7f3ca819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca819b330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800d6d0 tx=0x7f3c9800db90 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 shutdown_connections 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca8108d20 0x7f3ca819bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3c900b6610 0x7f3c900b8b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca81083d0 0x7f3ca819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 --2- 172.21.15.28:0/2889252536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ca8102ae0 0x7f3ca819b330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:33.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.663+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 >> 172.21.15.28:0/2889252536 conn(0x7f3ca80797b0 msgr2=0x7f3ca81044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:33.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.664+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 shutdown_connections 2024-09-21T05:44:33.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:33.664+0000 7f3cad27f640 1 -- 172.21.15.28:0/2889252536 wait complete. 2024-09-21T05:44:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2889252536' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2889252536' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:34.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:34.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.390 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-21T05:44:34.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2889252536' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:44:34.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:44:34.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:44:34.657 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 -- 172.21.15.28:0/4206394760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 msgr2=0x7f53ac10f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4206394760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac10f540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009ca0 tx=0x7f53a402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 -- 172.21.15.28:0/4206394760 shutdown_connections 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4206394760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac10f540 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4206394760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac1083c0 0x7f53ac1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4206394760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac102a80 0x7f53ac102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.111+0000 7f53b3a97640 1 -- 172.21.15.28:0/4206394760 >> 172.21.15.28:0/4206394760 conn(0x7f53ac079740 msgr2=0x7f53ac079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.112+0000 7f53b3a97640 1 -- 172.21.15.28:0/4206394760 shutdown_connections 2024-09-21T05:44:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.112+0000 7f53b3a97640 1 -- 172.21.15.28:0/4206394760 wait complete. 2024-09-21T05:44:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.112+0000 7f53b3a97640 1 Processor -- start 2024-09-21T05:44:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.112+0000 7f53b3a97640 1 -- start start 2024-09-21T05:44:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 0x7f53ac19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac1083c0 0x7f53ac19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac112980 con 0x7f53ac108d10 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac112800 con 0x7f53ac102a80 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b3a97640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac112b00 con 0x7f53ac1083c0 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b1833640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 0x7f53ac19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53582/0 (socket says 172.21.15.28:53582) 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b1833640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 0x7f53ac19b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55606/0 (socket says 172.21.15.28:55606) 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 -- 172.21.15.28:0/4052040035 learned_addr learned my addr 172.21.15.28:0/4052040035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b1032640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac1083c0 0x7f53ac19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 -- 172.21.15.28:0/4052040035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac1083c0 msgr2=0x7f53ac19b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac1083c0 0x7f53ac19b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 -- 172.21.15.28:0/4052040035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 msgr2=0x7f53ac19b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 0x7f53ac19b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 -- 172.21.15.28:0/4052040035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a64b0 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.113+0000 7f53b2034640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005cc0 tx=0x7f53a402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.114+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4003bf0 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.114+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a6740 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.114+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a4003d90 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.114+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a402fbc0 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.114+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a6c80 con 0x7f53ac108d10 2024-09-21T05:44:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.115+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ac103d00 con 0x7f53ac108d10 2024-09-21T05:44:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.116+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f53a402fd60 con 0x7f53ac108d10 2024-09-21T05:44:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.117+0000 7f53a2ffd640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 0x7f53980b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.117+0000 7f53b1833640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 0x7f53980b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.117+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 7342+0+0 (secure 0 0 0) 0x7f53a4108680 con 0x7f53ac108d10 2024-09-21T05:44:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.117+0000 7f53b1833640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 0x7f53980b8a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06c6c0 tx=0x7f53a800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:35.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.120+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f53a40cf240 con 0x7f53ac108d10 2024-09-21T05:44:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:35.308+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f53ac06bf70 con 0x7f53ac108d10 2024-09-21T05:44:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.088+0000 7f53a2ffd640 1 -- 172.21.15.28:0/4052040035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) ==== 138+0+0 (secure 0 0 0) 0x7f53ac06bf70 con 0x7f53ac108d10 2024-09-21T05:44:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-21T05:44:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 msgr2=0x7f53980b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 0x7f53980b8a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06c6c0 tx=0x7f53a800b390 comp rx=0 tx=0).stop 2024-09-21T05:44:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 msgr2=0x7f53ac19bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005cc0 tx=0x7f53a402f380 comp rx=0 tx=0).stop 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 shutdown_connections 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f53980b6560 0x7f53980b8a70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac108d10 0x7f53ac19bc70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53ac1083c0 0x7f53ac19b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 --2- 172.21.15.28:0/4052040035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53ac102a80 0x7f53ac19b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 >> 172.21.15.28:0/4052040035 conn(0x7f53ac079740 msgr2=0x7f53ac104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 shutdown_connections 2024-09-21T05:44:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:36.091+0000 7f53b3a97640 1 -- 172.21.15.28:0/4052040035 wait complete. 2024-09-21T05:44:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:36 smithi129 ceph-mon[34885]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:36 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T05:44:36.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:36 smithi028 ceph-mon[31227]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:36.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T05:44:36.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:36 smithi181 ceph-mon[34819]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:36.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:36 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T05:44:36.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-21T05:44:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:37.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:37 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T05:44:37.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:37 smithi129 ceph-mon[34885]: osdmap e67: 12 total, 12 up, 12 in 2024-09-21T05:44:37.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:37 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T05:44:37.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:37 smithi028 ceph-mon[31227]: osdmap e67: 12 total, 12 up, 12 in 2024-09-21T05:44:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 -- 172.21.15.28:0/2121403000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 msgr2=0x7fbcd0102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/2121403000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0102650 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009900 tx=0x7fbccc02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 -- 172.21.15.28:0/2121403000 shutdown_connections 2024-09-21T05:44:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/2121403000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd010ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/2121403000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 0x7fbcd0107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:37.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/2121403000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0102650 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:37.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 -- 172.21.15.28:0/2121403000 >> 172.21.15.28:0/2121403000 conn(0x7fbcd0079f90 msgr2=0x7fbcd007a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:37.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 -- 172.21.15.28:0/2121403000 shutdown_connections 2024-09-21T05:44:37.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.506+0000 7fbcd78ec640 1 -- 172.21.15.28:0/2121403000 wait complete. 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 Processor -- start 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 -- start start 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 0x7fbcd0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd0112210 con 0x7fbcd0102240 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd0112090 con 0x7fbcd01084d0 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd78ec640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd0112390 con 0x7fbcd0107b80 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd5688640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd5688640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33780/0 (socket says 172.21.15.28:33780) 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd5688640 1 -- 172.21.15.28:0/3740992890 learned_addr learned my addr 172.21.15.28:0/3740992890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:37.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd4e87640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 0x7fbcd0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.507+0000 7fbcd5e89640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 -- 172.21.15.28:0/3740992890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 msgr2=0x7fbcd0199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 0x7fbcd0199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 -- 172.21.15.28:0/3740992890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 msgr2=0x7fbcd019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 -- 172.21.15.28:0/3740992890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd019f480 con 0x7fbcd0102240 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5e89640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd5688640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005940 tx=0x7fbccc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc03f030 con 0x7fbcd0102240 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbccc003cb0 con 0x7fbcd0102240 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd019f710 con 0x7fbcd0102240 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc035070 con 0x7fbcd0102240 2024-09-21T05:44:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.508+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd01a5ed0 con 0x7fbcd0102240 2024-09-21T05:44:37.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.509+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd01034c0 con 0x7fbcd0102240 2024-09-21T05:44:37.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.511+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fbccc042030 con 0x7fbcd0102240 2024-09-21T05:44:37.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.512+0000 7fbcc67fc640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 0x7fbcbc0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.512+0000 7fbcd4e87640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 0x7fbcbc0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.512+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 7712+0+0 (secure 0 0 0) 0x7fbccc107c90 con 0x7fbcd0102240 2024-09-21T05:44:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.512+0000 7fbcd4e87640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 0x7fbcbc0b8af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd007a590 tx=0x7fbcc000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.515+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbccc0ce700 con 0x7fbcd0102240 2024-09-21T05:44:37.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:37 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4052040035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T05:44:37.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:37 smithi181 ceph-mon[34819]: osdmap e67: 12 total, 12 up, 12 in 2024-09-21T05:44:37.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:37.707+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbcd0102650 con 0x7fbcd0102240 2024-09-21T05:44:38.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.096+0000 7fbcc67fc640 1 -- 172.21.15.28:0/3740992890 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v69) ==== 170+0+0 (secure 0 0 0) 0x7fbcd0102650 con 0x7fbcd0102240 2024-09-21T05:44:38.098 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 msgr2=0x7fbcbc0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 0x7fbcbc0b8af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd007a590 tx=0x7fbcc000b390 comp rx=0 tx=0).stop 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 msgr2=0x7fbcd0199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbccc005940 tx=0x7fbccc0042f0 comp rx=0 tx=0).stop 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 shutdown_connections 2024-09-21T05:44:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd01084d0 0x7fbcd019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcd0107b80 0x7fbcd0199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcbc0b65e0 0x7fbcbc0b8af0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 --2- 172.21.15.28:0/3740992890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcd0102240 0x7fbcd0199510 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.099+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 >> 172.21.15.28:0/3740992890 conn(0x7fbcd0079f90 msgr2=0x7fbcd0100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.100+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 shutdown_connections 2024-09-21T05:44:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:38.100+0000 7fbcd78ec640 1 -- 172.21.15.28:0/3740992890 wait complete. 2024-09-21T05:44:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:38 smithi129 ceph-mon[34885]: pgmap v247: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:38 smithi129 ceph-mon[34885]: osdmap e68: 12 total, 12 up, 12 in 2024-09-21T05:44:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T05:44:38.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:38 smithi028 ceph-mon[31227]: pgmap v247: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:38.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:38 smithi028 ceph-mon[31227]: osdmap e68: 12 total, 12 up, 12 in 2024-09-21T05:44:38.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T05:44:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:38 smithi181 ceph-mon[34819]: pgmap v247: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:38 smithi181 ceph-mon[34819]: osdmap e68: 12 total, 12 up, 12 in 2024-09-21T05:44:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T05:44:38.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-21T05:44:39.040 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:39.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:39 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T05:44:39.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:39 smithi028 ceph-mon[31227]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T05:44:39.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:39 smithi028 ceph-mon[31227]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T05:44:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 -- 172.21.15.28:0/2932762183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7181083e0 msgr2=0x7fc7181087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/2932762183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7181083e0 0x7fc7181087f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009900 tx=0x7fc70c02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 -- 172.21.15.28:0/2932762183 shutdown_connections 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/2932762183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc718108d30 0x7fc71810f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/2932762183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7181083e0 0x7fc7181087f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/2932762183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718102aa0 0x7fc718102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 -- 172.21.15.28:0/2932762183 >> 172.21.15.28:0/2932762183 conn(0x7fc718079750 msgr2=0x7fc718079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 -- 172.21.15.28:0/2932762183 shutdown_connections 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.524+0000 7fc71ed1c640 1 -- 172.21.15.28:0/2932762183 wait complete. 2024-09-21T05:44:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 Processor -- start 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 -- start start 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7181083e0 0x7fc71819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 0x7fc71819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc718112a90 con 0x7fc718102aa0 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc718112910 con 0x7fc7181083e0 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71ed1c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc718112c10 con 0x7fc718108d30 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71cab8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71cab8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33810/0 (socket says 172.21.15.28:33810) 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71cab8640 1 -- 172.21.15.28:0/1378521908 learned_addr learned my addr 172.21.15.28:0/1378521908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc717fff640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7181083e0 0x7fc71819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.525+0000 7fc71d2b9640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 0x7fc71819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 -- 172.21.15.28:0/1378521908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 msgr2=0x7fc71819fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 0x7fc71819fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 -- 172.21.15.28:0/1378521908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7181083e0 msgr2=0x7fc71819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7181083e0 0x7fc71819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 -- 172.21.15.28:0/1378521908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7181a0620 con 0x7fc718102aa0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71d2b9640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 0x7fc71819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71cab8640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b4a0 tx=0x7fc70800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708014070 con 0x7fc718102aa0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc708010ac0 con 0x7fc718102aa0 2024-09-21T05:44:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708017c30 con 0x7fc718102aa0 2024-09-21T05:44:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.526+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7181a0910 con 0x7fc718102aa0 2024-09-21T05:44:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.527+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7181a7070 con 0x7fc718102aa0 2024-09-21T05:44:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.528+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc718102f30 con 0x7fc718102aa0 2024-09-21T05:44:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.529+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fc708017070 con 0x7fc718102aa0 2024-09-21T05:44:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.530+0000 7fc715ffb640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 0x7fc7000b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.530+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 8094+0+0 (secure 0 0 0) 0x7fc7080e6950 con 0x7fc718102aa0 2024-09-21T05:44:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.530+0000 7fc717fff640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 0x7fc7000b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.531+0000 7fc717fff640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 0x7fc7000b8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009230 tx=0x7fc70c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.533+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc7080ad2c0 con 0x7fc718102aa0 2024-09-21T05:44:39.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:39 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T05:44:39.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:39 smithi181 ceph-mon[34819]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T05:44:39.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:39 smithi181 ceph-mon[34819]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T05:44:39.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:39 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3740992890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T05:44:39.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:39 smithi129 ceph-mon[34885]: osdmap e69: 12 total, 12 up, 12 in 2024-09-21T05:44:39.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:39 smithi129 ceph-mon[34885]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T05:44:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:39.760+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc71819b7e0 con 0x7fc718102aa0 2024-09-21T05:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.120+0000 7fc715ffb640 1 -- 172.21.15.28:0/1378521908 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fc7080b16a0 con 0x7fc718102aa0 2024-09-21T05:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-09-21T05:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 msgr2=0x7fc7000b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 0x7fc7000b8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009230 tx=0x7fc70c031400 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 msgr2=0x7fc71819a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b4a0 tx=0x7fc70800b960 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 shutdown_connections 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc718108d30 0x7fc71819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7181083e0 0x7fc71819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7000b65e0 0x7fc7000b8af0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 --2- 172.21.15.28:0/1378521908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc718102aa0 0x7fc71819a6b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 >> 172.21.15.28:0/1378521908 conn(0x7fc718079750 msgr2=0x7fc718101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 shutdown_connections 2024-09-21T05:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:40.124+0000 7fc71ed1c640 1 -- 172.21.15.28:0/1378521908 wait complete. 2024-09-21T05:44:40.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:40 smithi181 ceph-mon[34819]: pgmap v250: 97 pgs: 18 creating+peering, 67 unknown, 12 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:40 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T05:44:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:40 smithi181 ceph-mon[34819]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T05:44:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:40 smithi028 ceph-mon[31227]: pgmap v250: 97 pgs: 18 creating+peering, 67 unknown, 12 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:40 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T05:44:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:40 smithi028 ceph-mon[31227]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T05:44:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:40 smithi129 ceph-mon[34885]: pgmap v250: 97 pgs: 18 creating+peering, 67 unknown, 12 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:40 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T05:44:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:40 smithi129 ceph-mon[34885]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-21T05:44:41.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:44:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:41.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:41 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T05:44:41.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:41 smithi028 ceph-mon[31227]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T05:44:41.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:41 smithi028 ceph-mon[31227]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T05:44:41.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:41 smithi028 ceph-mon[31227]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T05:44:41.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:41 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T05:44:41.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:41 smithi181 ceph-mon[34819]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T05:44:41.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:41 smithi181 ceph-mon[34819]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T05:44:41.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:41 smithi181 ceph-mon[34819]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T05:44:41.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:41 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1378521908' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T05:44:41.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:41 smithi129 ceph-mon[34885]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T05:44:41.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:41 smithi129 ceph-mon[34885]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-21T05:44:41.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:41 smithi129 ceph-mon[34885]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-21T05:44:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 -- 172.21.15.28:0/1871653351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c109e40 msgr2=0x7ff70c110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 --2- 172.21.15.28:0/1871653351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c109e40 0x7ff70c110670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff704009ca0 tx=0x7ff70402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 -- 172.21.15.28:0/1871653351 shutdown_connections 2024-09-21T05:44:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 --2- 172.21.15.28:0/1871653351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c109e40 0x7ff70c110670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 --2- 172.21.15.28:0/1871653351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c1094f0 0x7ff70c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 --2- 172.21.15.28:0/1871653351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c103bb0 0x7ff70c103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 -- 172.21.15.28:0/1871653351 >> 172.21.15.28:0/1871653351 conn(0x7ff70c0ff600 msgr2=0x7ff70c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 -- 172.21.15.28:0/1871653351 shutdown_connections 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.746+0000 7ff711e08640 1 -- 172.21.15.28:0/1871653351 wait complete. 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 Processor -- start 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 -- start start 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 0x7ff70c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 0x7ff70c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff70c113b00 con 0x7ff70c103bb0 2024-09-21T05:44:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff70c113980 con 0x7ff70c1094f0 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff711e08640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff70c113c80 con 0x7ff70c109e40 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.747+0000 7ff70b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33828/0 (socket says 172.21.15.28:33828) 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 -- 172.21.15.28:0/3787549971 learned_addr learned my addr 172.21.15.28:0/3787549971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff702ffd640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 0x7ff70c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70bfff640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 0x7ff70c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 -- 172.21.15.28:0/3787549971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 msgr2=0x7ff70c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 0x7ff70c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 -- 172.21.15.28:0/3787549971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 msgr2=0x7ff70c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 0x7ff70c19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 -- 172.21.15.28:0/3787549971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70c1a03e0 con 0x7ff70c103bb0 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff702ffd640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 0x7ff70c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70bfff640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 0x7ff70c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff70b7fe640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00e9f0 tx=0x7ff6fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc00cd80 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70c1a06d0 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6fc0108e0 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc018980 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.748+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff70c1a6e30 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.749+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff70c103fc0 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.751+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7ff6fc010430 con 0x7ff70c103bb0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.752+0000 7ff7097fa640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 0x7ff6f80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.752+0000 7ff702ffd640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 0x7ff6f80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.752+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7ff6fc0e7cc0 con 0x7ff70c103bb0 2024-09-21T05:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.752+0000 7ff702ffd640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 0x7ff6f80b8a70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff70c107f60 tx=0x7ff6f4005e50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.755+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6fc0ae510 con 0x7ff70c103bb0 2024-09-21T05:44:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.942+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff70c19b5a0 con 0x7ff70c103bb0 2024-09-21T05:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.944+0000 7ff7097fa640 1 -- 172.21.15.28:0/3787549971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+22595 (secure 0 0 0) 0x7ff6fc0cfe20 con 0x7ff70c103bb0 2024-09-21T05:44:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":71,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:44:39.762020+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:44:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 msgr2=0x7ff6f80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 0x7ff6f80b8a70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff70c107f60 tx=0x7ff6f4005e50 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 msgr2=0x7ff70c19a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00e9f0 tx=0x7ff6fc00eeb0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 shutdown_connections 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.946+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff70c109e40 0x7ff70c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff70c1094f0 0x7ff70c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff6f80b6560 0x7ff6f80b8a70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 --2- 172.21.15.28:0/3787549971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70c103bb0 0x7ff70c19a470 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 >> 172.21.15.28:0/3787549971 conn(0x7ff70c0ff600 msgr2=0x7ff70c101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 shutdown_connections 2024-09-21T05:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:41.947+0000 7ff711e08640 1 -- 172.21.15.28:0/3787549971 wait complete. 2024-09-21T05:44:42.044 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: 2024-09-21T05:44:41.761+0000 7fde2771d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-21T05:44:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:42 smithi028 ceph-mon[31227]: pgmap v253: 97 pgs: 20 creating+peering, 62 unknown, 15 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:42 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:active 2024-09-21T05:44:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:42 smithi028 ceph-mon[31227]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:42 smithi028 ceph-mon[31227]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:42 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3787549971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:42 smithi181 ceph-mon[34819]: pgmap v253: 97 pgs: 20 creating+peering, 62 unknown, 15 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:42 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:active 2024-09-21T05:44:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:42 smithi181 ceph-mon[34819]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:42 smithi181 ceph-mon[34819]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:42 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3787549971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:42.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:44:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:42 smithi129 ceph-mon[34885]: pgmap v253: 97 pgs: 20 creating+peering, 62 unknown, 15 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T05:44:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:42 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] up:active 2024-09-21T05:44:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:42 smithi129 ceph-mon[34885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:42 smithi129 ceph-mon[34885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:42 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3787549971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 -- 172.21.15.28:0/4104560914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c1083d0 msgr2=0x7f770c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 --2- 172.21.15.28:0/4104560914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c1083d0 0x7f770c1087e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7700009900 tx=0x7f770002a410 comp rx=0 tx=0).stop 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 -- 172.21.15.28:0/4104560914 shutdown_connections 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 --2- 172.21.15.28:0/4104560914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c108d20 0x7f770c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 --2- 172.21.15.28:0/4104560914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c1083d0 0x7f770c1087e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 --2- 172.21.15.28:0/4104560914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c102ae0 0x7f770c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 -- 172.21.15.28:0/4104560914 >> 172.21.15.28:0/4104560914 conn(0x7f770c0797b0 msgr2=0x7f770c079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:43.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 -- 172.21.15.28:0/4104560914 shutdown_connections 2024-09-21T05:44:43.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.297+0000 7f7713030640 1 -- 172.21.15.28:0/4104560914 wait complete. 2024-09-21T05:44:43.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 Processor -- start 2024-09-21T05:44:43.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 -- start start 2024-09-21T05:44:43.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:43.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c1083d0 0x7f770c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 0x7f770c1a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1129e0 con 0x7f770c102ae0 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f770c112860 con 0x7f770c1083d0 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7713030640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f770c112b60 con 0x7f770c108d20 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7710dcc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7710dcc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33850/0 (socket says 172.21.15.28:33850) 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7710dcc640 1 -- 172.21.15.28:0/1198360821 learned_addr learned my addr 172.21.15.28:0/1198360821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f770bfff640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c1083d0 0x7f770c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f77115cd640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 0x7f770c1a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7710dcc640 1 -- 172.21.15.28:0/1198360821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 msgr2=0x7f770c1a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.298+0000 7f7710dcc640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 0x7f770c1a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7710dcc640 1 -- 172.21.15.28:0/1198360821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c1083d0 msgr2=0x7f770c19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7710dcc640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c1083d0 0x7f770c19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7710dcc640 1 -- 172.21.15.28:0/1198360821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c1a2580 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f77115cd640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 0x7f770c1a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7710dcc640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00c890 tx=0x7f76fc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc004220 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c1a2870 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76fc0043c0 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc004a00 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.299+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a9090 con 0x7f770c102ae0 2024-09-21T05:44:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.300+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770c102ef0 con 0x7f770c102ae0 2024-09-21T05:44:43.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.302+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f76fc0079b0 con 0x7f770c102ae0 2024-09-21T05:44:43.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.303+0000 7f7709ffb640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 0x7f76f40b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.303+0000 7f770bfff640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 0x7f76f40b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.303+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f76fc0e7560 con 0x7f770c102ae0 2024-09-21T05:44:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.303+0000 7f770bfff640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 0x7f76f40b8aa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7700009230 tx=0x7f7700031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:43.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.306+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f76fc014030 con 0x7f770c102ae0 2024-09-21T05:44:43.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.492+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f770c19d800 con 0x7f770c102ae0 2024-09-21T05:44:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.494+0000 7f7709ffb640 1 -- 172.21.15.28:0/1198360821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+22595 (secure 0 0 0) 0x7f76fc0addb0 con 0x7f770c102ae0 2024-09-21T05:44:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:44:43.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":71,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:44:39.762020+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:44:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 msgr2=0x7f76f40b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 0x7f76f40b8aa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7700009230 tx=0x7f7700031400 comp rx=0 tx=0).stop 2024-09-21T05:44:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 msgr2=0x7f770c19c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00c890 tx=0x7f76fc00cd50 comp rx=0 tx=0).stop 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 shutdown_connections 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f770c108d20 0x7f770c1a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f770c1083d0 0x7f770c19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76f40b6590 0x7f76f40b8aa0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 --2- 172.21.15.28:0/1198360821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c102ae0 0x7f770c19c6d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 >> 172.21.15.28:0/1198360821 conn(0x7f770c0797b0 msgr2=0x7f770c06ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 shutdown_connections 2024-09-21T05:44:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:43.496+0000 7f7713030640 1 -- 172.21.15.28:0/1198360821 wait complete. 2024-09-21T05:44:44.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs set cephfs standby_count_wanted 0 2024-09-21T05:44:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:44.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:44 smithi181 ceph-mon[34819]: pgmap v254: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-21T05:44:44.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:44 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1198360821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:44 smithi028 ceph-mon[31227]: pgmap v254: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-21T05:44:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:44 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1198360821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:44 smithi129 ceph-mon[34885]: pgmap v254: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-09-21T05:44:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:44 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1198360821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:44:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.914+0000 7faacbb07640 1 -- 172.21.15.28:0/1567448271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 msgr2=0x7faac4110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.914+0000 7faacbb07640 1 --2- 172.21.15.28:0/1567448271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac4110640 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faabc009ca0 tx=0x7faabc02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 -- 172.21.15.28:0/1567448271 shutdown_connections 2024-09-21T05:44:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 --2- 172.21.15.28:0/1567448271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac4110640 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 --2- 172.21.15.28:0/1567448271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac41094c0 0x7faac41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 --2- 172.21.15.28:0/1567448271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac4103b80 0x7faac4103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 -- 172.21.15.28:0/1567448271 >> 172.21.15.28:0/1567448271 conn(0x7faac40ff5d0 msgr2=0x7faac4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 -- 172.21.15.28:0/1567448271 shutdown_connections 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 -- 172.21.15.28:0/1567448271 wait complete. 2024-09-21T05:44:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 Processor -- start 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.915+0000 7faacbb07640 1 -- start start 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac4103b80 0x7faac419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac41094c0 0x7faac419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faac4113ad0 con 0x7faac4109e10 2024-09-21T05:44:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faac4113950 con 0x7faac4103b80 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faacbb07640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faac4113c50 con 0x7faac41094c0 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33860/0 (socket says 172.21.15.28:33860) 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 -- 172.21.15.28:0/2742228255 learned_addr learned my addr 172.21.15.28:0/2742228255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faac98a3640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac4103b80 0x7faac419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faac90a2640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac41094c0 0x7faac419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 -- 172.21.15.28:0/2742228255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac41094c0 msgr2=0x7faac419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac41094c0 0x7faac419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 -- 172.21.15.28:0/2742228255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac4103b80 msgr2=0x7faac419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac4103b80 0x7faac419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 -- 172.21.15.28:0/2742228255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac41a03b0 con 0x7faac4109e10 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faaca0a4640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faabc005cc0 tx=0x7faabc02f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.916+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc003cf0 con 0x7faac4109e10 2024-09-21T05:44:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.917+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac41a0640 con 0x7faac4109e10 2024-09-21T05:44:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.917+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faabc0044f0 con 0x7faac4109e10 2024-09-21T05:44:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.917+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc02fd10 con 0x7faac4109e10 2024-09-21T05:44:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.917+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac41a6e00 con 0x7faac4109e10 2024-09-21T05:44:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.918+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac4103f90 con 0x7faac4109e10 2024-09-21T05:44:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.919+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7faabc004040 con 0x7faac4109e10 2024-09-21T05:44:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.921+0000 7faabaffd640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 0x7faab00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.921+0000 7faac98a3640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 0x7faab00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.921+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7faabc108a30 con 0x7faac4109e10 2024-09-21T05:44:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.921+0000 7faac98a3640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 0x7faab00b8a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faac00058c0 tx=0x7faac000a250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:44.924+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faabc0cf370 con 0x7faac4109e10 2024-09-21T05:44:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:45.151+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7faac419b570 con 0x7faac4109e10 2024-09-21T05:44:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.150+0000 7faabaffd640 1 -- 172.21.15.28:0/2742228255 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7faabc0d3750 con 0x7faac4109e10 2024-09-21T05:44:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 msgr2=0x7faab00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 0x7faab00b8a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faac00058c0 tx=0x7faac000a250 comp rx=0 tx=0).stop 2024-09-21T05:44:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 msgr2=0x7faac419fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faabc005cc0 tx=0x7faabc02f470 comp rx=0 tx=0).stop 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 shutdown_connections 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7faab00b6560 0x7faab00b8a70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4109e10 0x7faac419fc80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac41094c0 0x7faac419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 --2- 172.21.15.28:0/2742228255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac4103b80 0x7faac419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.153+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 >> 172.21.15.28:0/2742228255 conn(0x7faac40ff5d0 msgr2=0x7faac41019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.154+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 shutdown_connections 2024-09-21T05:44:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:46.154+0000 7faacbb07640 1 -- 172.21.15.28:0/2742228255 wait complete. 2024-09-21T05:44:46.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:46 smithi028 ceph-mon[31227]: pgmap v255: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-21T05:44:46.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:46 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T05:44:46.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:46 smithi181 ceph-mon[34819]: pgmap v255: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-21T05:44:46.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:46 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T05:44:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:46 smithi129 ceph-mon[34885]: pgmap v255: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-21T05:44:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:46 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T05:44:46.854 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-21T05:44:46.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs set cephfs max_mds 5 2024-09-21T05:44:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:47.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:47 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T05:44:47.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:47 smithi028 ceph-mon[31227]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:47.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T05:44:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:47 smithi181 ceph-mon[34819]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:47 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2742228255' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T05:44:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:47 smithi129 ceph-mon[34885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 -- 172.21.15.28:0/3498305153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102b80 msgr2=0x7f75cc1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3498305153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102b80 0x7f75cc1080b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009ca0 tx=0x7f75c402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 -- 172.21.15.28:0/3498305153 shutdown_connections 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3498305153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102b80 0x7f75cc1080b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3498305153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102770 0x7f75cc107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3498305153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 0x7f75cc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.614+0000 7f75d34ae640 1 -- 172.21.15.28:0/3498305153 >> 172.21.15.28:0/3498305153 conn(0x7f75cc079e40 msgr2=0x7f75cc07a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.615+0000 7f75d34ae640 1 -- 172.21.15.28:0/3498305153 shutdown_connections 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.615+0000 7f75d34ae640 1 -- 172.21.15.28:0/3498305153 wait complete. 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.615+0000 7f75d34ae640 1 Processor -- start 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.615+0000 7f75d34ae640 1 -- start start 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 0x7f75cc19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 0x7f75cc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc113ae0 con 0x7f75cc102770 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc113960 con 0x7f75cc06b2f0 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc113c60 con 0x7f75cc102b80 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37836/0 (socket says 172.21.15.28:37836) 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 -- 172.21.15.28:0/3678761450 learned_addr learned my addr 172.21.15.28:0/3678761450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d124a640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 0x7f75cc19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d1a4b640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 0x7f75cc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 -- 172.21.15.28:0/3678761450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 msgr2=0x7f75cc19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 0x7f75cc19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 -- 172.21.15.28:0/3678761450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 msgr2=0x7f75cc19a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 0x7f75cc19a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 -- 172.21.15.28:0/3678761450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc1a03b0 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d1a4b640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 0x7f75cc19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d0a49640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00d740 tx=0x7f75bc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc017070 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75bc00de50 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75cc1a06a0 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc00f620 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.616+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75cc1a6e00 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.618+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75cc102230 con 0x7f75cc102770 2024-09-21T05:44:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.619+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f75bc01a070 con 0x7f75cc102770 2024-09-21T05:44:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.620+0000 7f75c27fc640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 0x7f75b80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.621+0000 7f75d124a640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 0x7f75b80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.621+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f75bc0e6d30 con 0x7f75cc102770 2024-09-21T05:44:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.621+0000 7f75d124a640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 0x7f75b80b8af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005fd0 tx=0x7f75c8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:47.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.624+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f75bc0ad660 con 0x7f75cc102770 2024-09-21T05:44:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:47.853+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f75cc1a7130 con 0x7f75cc102770 2024-09-21T05:44:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.162+0000 7f75c27fc640 1 -- 172.21.15.28:0/3678761450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f75bc0b1a40 con 0x7f75cc102770 2024-09-21T05:44:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 msgr2=0x7f75b80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 0x7f75b80b8af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005fd0 tx=0x7f75c8005850 comp rx=0 tx=0).stop 2024-09-21T05:44:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 msgr2=0x7f75cc19a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00d740 tx=0x7f75bc00dc00 comp rx=0 tx=0).stop 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 shutdown_connections 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75cc102b80 0x7f75cc19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f75b80b65e0 0x7f75b80b8af0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75cc102770 0x7f75cc19a9e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 --2- 172.21.15.28:0/3678761450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75cc06b2f0 0x7f75cc19a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 >> 172.21.15.28:0/3678761450 conn(0x7f75cc079e40 msgr2=0x7f75cc104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 shutdown_connections 2024-09-21T05:44:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:48.165+0000 7f75d34ae640 1 -- 172.21.15.28:0/3678761450 wait complete. 2024-09-21T05:44:48.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:48 smithi028 ceph-mon[31227]: pgmap v256: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T05:44:48.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:48 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T05:44:48.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:48 smithi181 ceph-mon[34819]: pgmap v256: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T05:44:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:48 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T05:44:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:48 smithi129 ceph-mon[34885]: pgmap v256: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T05:44:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:48 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T05:44:48.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs set cephfs allow_standby_replay true 2024-09-21T05:44:49.061 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T05:44:49.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:49 smithi028 ceph-mon[31227]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T05:44:49.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T05:44:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T05:44:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T05:44:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T05:44:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:49 smithi181 ceph-mon[34819]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 -- 172.21.15.28:0/564300552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f80102ae0 msgr2=0x7f1f80102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 --2- 172.21.15.28:0/564300552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f80102ae0 0x7f1f80102ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009900 tx=0x7f1f7c02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 -- 172.21.15.28:0/564300552 shutdown_connections 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 --2- 172.21.15.28:0/564300552 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80108d20 0x7f1f8010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 --2- 172.21.15.28:0/564300552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f801083d0 0x7f1f801087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:49.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 --2- 172.21.15.28:0/564300552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f80102ae0 0x7f1f80102ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 -- 172.21.15.28:0/564300552 >> 172.21.15.28:0/564300552 conn(0x7f1f80079760 msgr2=0x7f1f80079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 -- 172.21.15.28:0/564300552 shutdown_connections 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.531+0000 7f1f872e2640 1 -- 172.21.15.28:0/564300552 wait complete. 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 Processor -- start 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 -- start start 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80102ae0 0x7f1f8019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f80108d20 0x7f1f8019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112990 con 0x7f1f801083d0 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112810 con 0x7f1f80108d20 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f872e2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112b10 con 0x7f1f80102ae0 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37858/0 (socket says 172.21.15.28:37858) 2024-09-21T05:44:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 -- 172.21.15.28:0/1879398599 learned_addr learned my addr 172.21.15.28:0/1879398599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8507e640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80102ae0 0x7f1f8019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8587f640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f80108d20 0x7f1f8019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 -- 172.21.15.28:0/1879398599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80102ae0 msgr2=0x7f1f8019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80102ae0 0x7f1f8019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 -- 172.21.15.28:0/1879398599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f80108d20 msgr2=0x7f1f8019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f80108d20 0x7f1f8019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.532+0000 7f1f8487d640 1 -- 172.21.15.28:0/1879398599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f801a0360 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f8487d640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c970 tx=0x7f1f7000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70014070 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f70004440 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f801a0650 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70003e50 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.533+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f801a6e70 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.534+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f80103d10 con 0x7f1f801083d0 2024-09-21T05:44:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.535+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f1f70017070 con 0x7f1f801083d0 2024-09-21T05:44:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.536+0000 7f1f767fc640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 0x7f1f6c0b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.537+0000 7f1f8507e640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 0x7f1f6c0b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.537+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f1f700e7570 con 0x7f1f801083d0 2024-09-21T05:44:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.537+0000 7f1f8507e640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 0x7f1f6c0b8cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c031440 tx=0x7f1f7c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.540+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1f7001e030 con 0x7f1f801083d0 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3678761450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-21T05:44:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:49 smithi129 ceph-mon[34885]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-21T05:44:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:49.767+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f1f80102ef0 con 0x7f1f801083d0 2024-09-21T05:44:50.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.178+0000 7f1f767fc640 1 -- 172.21.15.28:0/1879398599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f1f80102ef0 con 0x7f1f801083d0 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 msgr2=0x7f1f6c0b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 0x7f1f6c0b8cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c031440 tx=0x7f1f7c009560 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 msgr2=0x7f1f8019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c970 tx=0x7f1f7000ce30 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 shutdown_connections 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f80108d20 0x7f1f8019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1f6c0b67b0 0x7f1f6c0b8cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f801083d0 0x7f1f8019aa50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 --2- 172.21.15.28:0/1879398599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f80102ae0 0x7f1f8019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 >> 172.21.15.28:0/1879398599 conn(0x7f1f80079760 msgr2=0x7f1f801019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 shutdown_connections 2024-09-21T05:44:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:50.181+0000 7f1f872e2640 1 -- 172.21.15.28:0/1879398599 wait complete. 2024-09-21T05:44:50.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-09-21T05:44:50.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:active 2024-09-21T05:44:50.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T05:44:50.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T05:44:50.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T05:44:50.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T05:44:50.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:50 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:active 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T05:44:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:50 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] up:active 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-21T05:44:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:50 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T05:44:50.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs set cephfs session_timeout 300 2024-09-21T05:44:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:active 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T05:44:51.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:51 smithi028 ceph-mon[31227]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:active 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T05:44:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-mon[34819]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T05:44:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 -- 172.21.15.28:0/319025696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc104840 msgr2=0x7fa8bc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 --2- 172.21.15.28:0/319025696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc104840 0x7fa8bc104c50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009900 tx=0x7fa8b802a410 comp rx=0 tx=0).stop 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 -- 172.21.15.28:0/319025696 shutdown_connections 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 --2- 172.21.15.28:0/319025696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc10ddf0 0x7fa8bc110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 --2- 172.21.15.28:0/319025696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc105270 0x7fa8bc10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 --2- 172.21.15.28:0/319025696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc104840 0x7fa8bc104c50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 -- 172.21.15.28:0/319025696 >> 172.21.15.28:0/319025696 conn(0x7fa8bc079730 msgr2=0x7fa8bc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 -- 172.21.15.28:0/319025696 shutdown_connections 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.584+0000 7fa8c432d640 1 -- 172.21.15.28:0/319025696 wait complete. 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 Processor -- start 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 -- start start 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc104840 0x7fa8bc19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc10ddf0 0x7fa8bc19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113a90 con 0x7fa8bc105270 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113910 con 0x7fa8bc104840 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c432d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113c10 con 0x7fa8bc10ddf0 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.585+0000 7fa8c18c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37880/0 (socket says 172.21.15.28:37880) 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 -- 172.21.15.28:0/2225970131 learned_addr learned my addr 172.21.15.28:0/2225970131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c20c9640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc104840 0x7fa8bc19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c28ca640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc10ddf0 0x7fa8bc19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 -- 172.21.15.28:0/2225970131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc10ddf0 msgr2=0x7fa8bc19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc10ddf0 0x7fa8bc19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 -- 172.21.15.28:0/2225970131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc104840 msgr2=0x7fa8bc19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc104840 0x7fa8bc19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 -- 172.21.15.28:0/2225970131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc1a0430 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c18c8640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c970 tx=0x7fa8ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac004220 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0043c0 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac004a00 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc1a0720 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.586+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a6e80 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.588+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8bc104c50 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.589+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fa8ac0079b0 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.590+0000 7fa8b37fe640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 0x7fa8a80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.590+0000 7fa8c20c9640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 0x7fa8a80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.590+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7fa8ac0e7560 con 0x7fa8bc105270 2024-09-21T05:44:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.590+0000 7fa8c20c9640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 0x7fa8a80b8af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b8031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:51.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.593+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa8ac014030 con 0x7fa8bc105270 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1879398599' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] up:active 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-21T05:44:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:51 smithi129 ceph-mon[34885]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-21T05:44:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:51.823+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa8bc19b5f0 con 0x7fa8bc105270 2024-09-21T05:44:52.044 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:44:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: 2024-09-21T05:44:51.762+0000 7fde2771d640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-21T05:44:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.047+0000 7fa8b37fe640 1 -- 172.21.15.28:0/2225970131 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7fa8bc19b5f0 con 0x7fa8bc105270 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 msgr2=0x7fa8a80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 0x7fa8a80b8af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b8031400 comp rx=0 tx=0).stop 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 msgr2=0x7fa8bc19aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c970 tx=0x7fa8ac00ce30 comp rx=0 tx=0).stop 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 shutdown_connections 2024-09-21T05:44:52.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8bc10ddf0 0x7fa8bc19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa8a80b65e0 0x7fa8a80b8af0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc105270 0x7fa8bc19aa60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 --2- 172.21.15.28:0/2225970131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8bc104840 0x7fa8bc19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 >> 172.21.15.28:0/2225970131 conn(0x7fa8bc079730 msgr2=0x7fa8bc107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 shutdown_connections 2024-09-21T05:44:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:52.050+0000 7fa8c432d640 1 -- 172.21.15.28:0/2225970131 wait complete. 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 4 op/s 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:active 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: Cluster is now healthy 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T05:44:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T05:44:52.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T05:44:52.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T05:44:52.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:active 2024-09-21T05:44:52.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:52 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 4 op/s 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:active 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: Cluster is now healthy 2024-09-21T05:44:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:active 2024-09-21T05:44:52.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:52 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:44:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: pgmap v258: 97 pgs: 97 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 4 op/s 2024-09-21T05:44:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] up:active 2024-09-21T05:44:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T05:44:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: Cluster is now healthy 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2225970131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] up:active 2024-09-21T05:44:52.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:52 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:44:52.756 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-09-21T05:44:52.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs subvolumegroup create cephfs qa 2024-09-21T05:44:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 -- 172.21.15.28:0/1878055372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c1081b0 msgr2=0x7fc73c1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 --2- 172.21.15.28:0/1878055372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c1081b0 0x7fc73c1085c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc72402a410 comp rx=0 tx=0).stop 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 -- 172.21.15.28:0/1878055372 shutdown_connections 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 --2- 172.21.15.28:0/1878055372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c108b00 0x7fc73c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 --2- 172.21.15.28:0/1878055372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c1081b0 0x7fc73c1085c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 --2- 172.21.15.28:0/1878055372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 0x7fc73c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 -- 172.21.15.28:0/1878055372 >> 172.21.15.28:0/1878055372 conn(0x7fc73c0fe2c0 msgr2=0x7fc73c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 -- 172.21.15.28:0/1878055372 shutdown_connections 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.490+0000 7fc74159f640 1 -- 172.21.15.28:0/1878055372 wait complete. 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 Processor -- start 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 -- start start 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 0x7fc73c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c1081b0 0x7fc73c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c1127e0 con 0x7fc73c108b00 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c112660 con 0x7fc73c102870 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc74159f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c112960 con 0x7fc73c1081b0 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc740d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:53.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc740d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37886/0 (socket says 172.21.15.28:37886) 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.491+0000 7fc740d9e640 1 -- 172.21.15.28:0/415231039 learned_addr learned my addr 172.21.15.28:0/415231039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc73bfff640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 0x7fc73c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc733fff640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c1081b0 0x7fc73c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 -- 172.21.15.28:0/415231039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c1081b0 msgr2=0x7fc73c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c1081b0 0x7fc73c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 -- 172.21.15.28:0/415231039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 msgr2=0x7fc73c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 0x7fc73c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 -- 172.21.15.28:0/415231039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c1a0290 con 0x7fc73c108b00 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc740d9e640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b970 tx=0x7fc73400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734019070 con 0x7fc73c108b00 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c1a0580 con 0x7fc73c108b00 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc734014440 con 0x7fc73c108b00 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7340075d0 con 0x7fc73c108b00 2024-09-21T05:44:53.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.492+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c1a6ce0 con 0x7fc73c108b00 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.493+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc73c103af0 con 0x7fc73c108b00 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.495+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7fc73401c070 con 0x7fc73c108b00 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.495+0000 7fc739ffb640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 0x7fc7280b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.496+0000 7fc73bfff640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 0x7fc7280b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.496+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7fc7340e6850 con 0x7fc73c108b00 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.496+0000 7fc73bfff640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 0x7fc7280b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00bbe0 tx=0x7fc72c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:53.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.500+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc7340b14e0 con 0x7fc73c108b00 2024-09-21T05:44:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.716+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc73c1070c0 con 0x7fc7280b6560 2024-09-21T05:44:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.744+0000 7fc739ffb640 1 -- 172.21.15.28:0/415231039 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc73c1070c0 con 0x7fc7280b6560 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 msgr2=0x7fc7280b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 0x7fc7280b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00bbe0 tx=0x7fc72c00b390 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 msgr2=0x7fc73c19fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b970 tx=0x7fc73400be30 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 shutdown_connections 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc7280b6560 0x7fc7280b8a70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc73c108b00 0x7fc73c19fb60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc73c1081b0 0x7fc73c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 --2- 172.21.15.28:0/415231039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c102870 0x7fc73c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 >> 172.21.15.28:0/415231039 conn(0x7fc73c0fe2c0 msgr2=0x7fc73c06b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 shutdown_connections 2024-09-21T05:44:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:53.747+0000 7fc74159f640 1 -- 172.21.15.28:0/415231039 wait complete. 2024-09-21T05:44:54.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:44:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:44:53.722+0000 7f3f7f0f2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:44:54.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:44:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:44:53.722+0000 7f3f7f0f2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:44:54.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:44:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:44:53.722+0000 7f3f7f0f2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:44:54.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:44:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:44:53.722+0000 7f3f7f0f2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:44:54.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:44:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:44:53.722+0000 7f3f7f0f2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:44:54.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-21T05:44:54.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:54 smithi181 ceph-mon[34819]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T05:44:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:54 smithi028 ceph-mon[31227]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T05:44:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:54 smithi129 ceph-mon[34885]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 -- 172.21.15.28:0/2299359314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3744105300 msgr2=0x7f374410d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 --2- 172.21.15.28:0/2299359314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3744105300 0x7f374410d850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3738009900 tx=0x7f373802a410 comp rx=0 tx=0).stop 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 -- 172.21.15.28:0/2299359314 shutdown_connections 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 --2- 172.21.15.28:0/2299359314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f374410de10 0x7f3744110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 --2- 172.21.15.28:0/2299359314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3744105300 0x7f374410d850 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 --2- 172.21.15.28:0/2299359314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f3744104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 -- 172.21.15.28:0/2299359314 >> 172.21.15.28:0/2299359314 conn(0x7f3744079730 msgr2=0x7f3744079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 -- 172.21.15.28:0/2299359314 shutdown_connections 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.015+0000 7f374bf95640 1 -- 172.21.15.28:0/2299359314 wait complete. 2024-09-21T05:44:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 Processor -- start 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 -- start start 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f374419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3744105300 0x7f374419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3744113a70 con 0x7f374410de10 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37441138f0 con 0x7f3744105300 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.016+0000 7f374bf95640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3744113bf0 con 0x7f37441049b0 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3749530640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3744105300 0x7f374419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37938/0 (socket says 172.21.15.28:37938) 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 -- 172.21.15.28:0/1735918188 learned_addr learned my addr 172.21.15.28:0/1735918188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3749d31640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f374419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 -- 172.21.15.28:0/1735918188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 msgr2=0x7f374419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f374419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 -- 172.21.15.28:0/1735918188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3744105300 msgr2=0x7f374419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3744105300 0x7f374419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 -- 172.21.15.28:0/1735918188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37441a04b0 con 0x7f374410de10 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3749d31640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f374419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f374a532640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f373c00d750 tx=0x7f373c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c017070 con 0x7f374410de10 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f373c0054d0 con 0x7f374410de10 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.017+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c004c60 con 0x7f374410de10 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.018+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37441a07a0 con 0x7f374410de10 2024-09-21T05:44:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.018+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37441a6f00 con 0x7f374410de10 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.019+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3744104dc0 con 0x7f374410de10 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.020+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 154529+0+0 (secure 0 0 0) 0x7f373c009bd0 con 0x7f374410de10 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.021+0000 7f3742ffd640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 0x7f372c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.021+0000 7f3749d31640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 0x7f372c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.021+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f373c0e6f30 con 0x7f374410de10 2024-09-21T05:44:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.022+0000 7f3749d31640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 0x7f372c0b8af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f374406bc60 tx=0x7f3734009670 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.025+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f373c0b1c40 con 0x7f374410de10 2024-09-21T05:44:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.215+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f373c0a9480 con 0x7f374410de10 2024-09-21T05:44:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.246+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3744108f70 con 0x7f372c0b65e0 2024-09-21T05:44:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.269+0000 7f3742ffd640 1 -- 172.21.15.28:0/1735918188 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3744108f70 con 0x7f372c0b65e0 2024-09-21T05:44:55.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:55 smithi028 ceph-mon[31227]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.271+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 msgr2=0x7f372c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 0x7f372c0b8af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f374406bc60 tx=0x7f3734009670 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 msgr2=0x7f374419fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f373c00d750 tx=0x7f373c00dc10 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 shutdown_connections 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f372c0b65e0 0x7f372c0b8af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374410de10 0x7f374419fd80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3744105300 0x7f374419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 --2- 172.21.15.28:0/1735918188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37441049b0 0x7f374419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 >> 172.21.15.28:0/1735918188 conn(0x7f3744079730 msgr2=0x7f3744100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 shutdown_connections 2024-09-21T05:44:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:55.272+0000 7f374bf95640 1 -- 172.21.15.28:0/1735918188 wait complete. 2024-09-21T05:44:55.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:55 smithi181 ceph-mon[34819]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:55 smithi129 ceph-mon[34885]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:55.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-21T05:44:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:56.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:56 smithi028 ceph-mon[31227]: pgmap v260: 97 pgs: 97 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T05:44:56.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:56 smithi028 ceph-mon[31227]: mgrmap e22: x(active, since 6m), standbys: z, y 2024-09-21T05:44:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:56 smithi028 ceph-mon[31227]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:56 smithi181 ceph-mon[34819]: pgmap v260: 97 pgs: 97 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T05:44:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:56 smithi181 ceph-mon[34819]: mgrmap e22: x(active, since 6m), standbys: z, y 2024-09-21T05:44:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:56 smithi181 ceph-mon[34819]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:56 smithi129 ceph-mon[34885]: pgmap v260: 97 pgs: 97 active+clean; 583 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-09-21T05:44:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:56 smithi129 ceph-mon[34885]: mgrmap e22: x(active, since 6m), standbys: z, y 2024-09-21T05:44:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:56 smithi129 ceph-mon[34885]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 -- 172.21.15.28:0/2798497478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650075730 msgr2=0x7f9650075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 --2- 172.21.15.28:0/2798497478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650075730 0x7f9650075b40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9640009900 tx=0x7f964002a3f0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 -- 172.21.15.28:0/2798497478 shutdown_connections 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 --2- 172.21.15.28:0/2798497478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 0x7f9650112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 --2- 172.21.15.28:0/2798497478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650076080 0x7f965010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 --2- 172.21.15.28:0/2798497478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650075730 0x7f9650075b40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 -- 172.21.15.28:0/2798497478 >> 172.21.15.28:0/2798497478 conn(0x7f965007b850 msgr2=0x7f965007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 -- 172.21.15.28:0/2798497478 shutdown_connections 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.617+0000 7f9656442640 1 -- 172.21.15.28:0/2798497478 wait complete. 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 Processor -- start 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 -- start start 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650075730 0x7f965019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 0x7f96501a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9650115c80 con 0x7f9650076080 2024-09-21T05:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9650115b00 con 0x7f9650075730 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f9656442640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9650115e00 con 0x7f9650110140 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58972/0 (socket says 172.21.15.28:58972) 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 -- 172.21.15.28:0/1999973946 learned_addr learned my addr 172.21.15.28:0/1999973946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f96549df640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 0x7f96501a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964ffff640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650075730 0x7f965019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 -- 172.21.15.28:0/1999973946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 msgr2=0x7f96501a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 0x7f96501a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 -- 172.21.15.28:0/1999973946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650075730 msgr2=0x7f965019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.618+0000 7f964f7fe640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650075730 0x7f965019c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f964f7fe640 1 -- 172.21.15.28:0/1999973946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96501a2520 con 0x7f9650076080 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f964f7fe640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f964400c8c0 tx=0x7f964400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644007bb0 con 0x7f9650076080 2024-09-21T05:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9644007d50 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96501a2810 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440187c0 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.619+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96501a9030 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.620+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9650075b40 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.621+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f964400fa20 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.622+0000 7f964d7fa640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 0x7f96380b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.622+0000 7f964ffff640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 0x7f96380b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.622+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f96440e78b0 con 0x7f9650076080 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.623+0000 7f964ffff640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 0x7f96380b8b30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9640005f70 tx=0x7f9640000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.625+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9644014030 con 0x7f9650076080 2024-09-21T05:44:56.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.842+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f965010a3f0 con 0x7f96380b6620 2024-09-21T05:44:56.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.848+0000 7f964d7fa640 1 -- 172.21.15.28:0/1999973946 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f965010a3f0 con 0x7f96380b6620 2024-09-21T05:44:56.850 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/a1368cb0-e874-40bb-8053-d7490bd59007 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 msgr2=0x7f96380b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 0x7f96380b8b30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9640005f70 tx=0x7f9640000bc0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 msgr2=0x7f965019cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f964400c8c0 tx=0x7f964400cd80 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 shutdown_connections 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9650110140 0x7f96501a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f96380b6620 0x7f96380b8b30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650076080 0x7f965019cc10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 --2- 172.21.15.28:0/1999973946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9650075730 0x7f965019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 >> 172.21.15.28:0/1999973946 conn(0x7f965007b850 msgr2=0x7f9650109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 shutdown_connections 2024-09-21T05:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:56.851+0000 7f9656442640 1 -- 172.21.15.28:0/1999973946 wait complete. 2024-09-21T05:44:57.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-21T05:44:57.706 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 -- 172.21.15.28:0/3166263855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f168810de10 msgr2=0x7f1688110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 --2- 172.21.15.28:0/3166263855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f168810de10 0x7f1688110300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1680009900 tx=0x7f168002a410 comp rx=0 tx=0).stop 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 -- 172.21.15.28:0/3166263855 shutdown_connections 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 --2- 172.21.15.28:0/3166263855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f168810de10 0x7f1688110300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 --2- 172.21.15.28:0/3166263855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1688105300 0x7f168810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 --2- 172.21.15.28:0/3166263855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16881049b0 0x7f1688104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 -- 172.21.15.28:0/3166263855 >> 172.21.15.28:0/3166263855 conn(0x7f1688079730 msgr2=0x7f1688079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 -- 172.21.15.28:0/3166263855 shutdown_connections 2024-09-21T05:44:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.164+0000 7f168fb39640 1 -- 172.21.15.28:0/3166263855 wait complete. 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 Processor -- start 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 -- start start 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 0x7f168819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 0x7f168819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1688113a70 con 0x7f16881049b0 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16881138f0 con 0x7f1688105300 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168fb39640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1688113bf0 con 0x7f168810de10 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168d8d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58982/0 (socket says 172.21.15.28:58982) 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168d8d5640 1 -- 172.21.15.28:0/3818507326 learned_addr learned my addr 172.21.15.28:0/3818507326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168d0d4640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 0x7f168819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.165+0000 7f168e0d6640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 0x7f168819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 -- 172.21.15.28:0/3818507326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 msgr2=0x7f168819fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 0x7f168819fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 -- 172.21.15.28:0/3818507326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 msgr2=0x7f168819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 0x7f168819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 -- 172.21.15.28:0/3818507326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16881a0350 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d0d4640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 0x7f168819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168e0d6640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 0x7f168819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168d8d5640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f167800ccb0 tx=0x7f167800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1678003b70 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f167800ae50 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16881a0640 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167800f6f0 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.166+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a6e60 con 0x7f16881049b0 2024-09-21T05:44:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.167+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1688104dc0 con 0x7f16881049b0 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.169+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f1678004000 con 0x7f16881049b0 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.169+0000 7f1686ffd640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 0x7f16700b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.170+0000 7f168d0d4640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 0x7f16700b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.170+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f16780e76b0 con 0x7f16881049b0 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.170+0000 7f168d0d4640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 0x7f16700b8b30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f167c005e00 tx=0x7f167c005c70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.174+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1678014070 con 0x7f16881049b0 2024-09-21T05:44:58.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.390+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1688063800 con 0x7f16700b6620 2024-09-21T05:44:58.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.417+0000 7f1686ffd640 1 -- 172.21.15.28:0/3818507326 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1688063800 con 0x7f16700b6620 2024-09-21T05:44:58.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:58 smithi028 ceph-mon[31227]: pgmap v261: 97 pgs: 97 active+clean; 585 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.1 KiB/s wr, 18 op/s 2024-09-21T05:44:58.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:58 smithi028 ceph-mon[31227]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:58.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 msgr2=0x7f16700b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:58.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 0x7f16700b8b30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f167c005e00 tx=0x7f167c005c70 comp rx=0 tx=0).stop 2024-09-21T05:44:58.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 msgr2=0x7f168819a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f167800ccb0 tx=0x7f167800a620 comp rx=0 tx=0).stop 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 shutdown_connections 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168810de10 0x7f168819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1688105300 0x7f168819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16700b6620 0x7f16700b8b30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 --2- 172.21.15.28:0/3818507326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16881049b0 0x7f168819a4a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.419+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 >> 172.21.15.28:0/3818507326 conn(0x7f1688079730 msgr2=0x7f16881007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.420+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 shutdown_connections 2024-09-21T05:44:58.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:58.420+0000 7f168fb39640 1 -- 172.21.15.28:0/3818507326 wait complete. 2024-09-21T05:44:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:58 smithi181 ceph-mon[34819]: pgmap v261: 97 pgs: 97 active+clean; 585 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.1 KiB/s wr, 18 op/s 2024-09-21T05:44:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:58 smithi181 ceph-mon[34819]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:58 smithi129 ceph-mon[34885]: pgmap v261: 97 pgs: 97 active+clean; 585 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.1 KiB/s wr, 18 op/s 2024-09-21T05:44:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:58 smithi129 ceph-mon[34885]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:59.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-21T05:44:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:44:59.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:44:59 smithi028 ceph-mon[31227]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:59.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:44:59 smithi181 ceph-mon[34819]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:44:59 smithi129 ceph-mon[34885]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 -- 172.21.15.28:0/336891819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c1094a0 msgr2=0x7f350c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 --2- 172.21.15.28:0/336891819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c1094a0 0x7f350c1098b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009900 tx=0x7f34fc02a410 comp rx=0 tx=0).stop 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 -- 172.21.15.28:0/336891819 shutdown_connections 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 --2- 172.21.15.28:0/336891819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c109df0 0x7f350c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 --2- 172.21.15.28:0/336891819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c1094a0 0x7f350c1098b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 --2- 172.21.15.28:0/336891819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 0x7f350c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.753+0000 7f35138c4640 1 -- 172.21.15.28:0/336891819 >> 172.21.15.28:0/336891819 conn(0x7f350c07b850 msgr2=0x7f350c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.754+0000 7f35138c4640 1 -- 172.21.15.28:0/336891819 shutdown_connections 2024-09-21T05:44:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.754+0000 7f35138c4640 1 -- 172.21.15.28:0/336891819 wait complete. 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.754+0000 7f35138c4640 1 Processor -- start 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.754+0000 7f35138c4640 1 -- start start 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 0x7f350c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c1094a0 0x7f350c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f350c113ad0 con 0x7f350c109df0 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f350c113950 con 0x7f350c103b60 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f35138c4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f350c113c50 con 0x7f350c1094a0 2024-09-21T05:44:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59002/0 (socket says 172.21.15.28:59002) 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 -- 172.21.15.28:0/4094679841 learned_addr learned my addr 172.21.15.28:0/4094679841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3510e5f640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c1094a0 0x7f350c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511660640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 0x7f350c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 -- 172.21.15.28:0/4094679841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c1094a0 msgr2=0x7f350c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c1094a0 0x7f350c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 -- 172.21.15.28:0/4094679841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 msgr2=0x7f350c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 0x7f350c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 -- 172.21.15.28:0/4094679841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c1a8820 con 0x7f350c109df0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.755+0000 7f3511e61640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f350400eaa0 tx=0x7f350400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.756+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504004d20 con 0x7f350c109df0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.756+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c1a8b10 con 0x7f350c109df0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.756+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3504005020 con 0x7f350c109df0 2024-09-21T05:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.756+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040116b0 con 0x7f350c109df0 2024-09-21T05:44:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.756+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c1a9050 con 0x7f350c109df0 2024-09-21T05:44:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.757+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c103f70 con 0x7f350c109df0 2024-09-21T05:44:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.758+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f35040051d0 con 0x7f350c109df0 2024-09-21T05:44:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.759+0000 7f35027fc640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 0x7f34f80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:44:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.759+0000 7f3511660640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 0x7f34f80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:44:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.759+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f35040e7700 con 0x7f350c109df0 2024-09-21T05:44:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.760+0000 7f3511660640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 0x7f34f80b8b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f350c077160 tx=0x7f3508009110 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:44:59.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.763+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3504016030 con 0x7f350c109df0 2024-09-21T05:44:59.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.980+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f350c063800 con 0x7f34f80b65f0 2024-09-21T05:44:59.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.985+0000 7f35027fc640 1 -- 172.21.15.28:0/4094679841 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f350c063800 con 0x7f34f80b65f0 2024-09-21T05:44:59.987 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/7e06d2f3-dff5-4680-8244-da51f271037c 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 msgr2=0x7f34f80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 0x7f34f80b8b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f350c077160 tx=0x7f3508009110 comp rx=0 tx=0).stop 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 msgr2=0x7f350c076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f350400eaa0 tx=0x7f350400eea0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 shutdown_connections 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f34f80b65f0 0x7f34f80b8b00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f350c109df0 0x7f350c076190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f350c1094a0 0x7f350c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 --2- 172.21.15.28:0/4094679841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f350c103b60 0x7f350c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:44:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 >> 172.21.15.28:0/4094679841 conn(0x7f350c07b850 msgr2=0x7f350c1075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:44:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 shutdown_connections 2024-09-21T05:44:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:44:59.988+0000 7f35138c4640 1 -- 172.21.15.28:0/4094679841 wait complete. 2024-09-21T05:45:00.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:00 smithi028 ceph-mon[31227]: pgmap v262: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-09-21T05:45:00.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:00 smithi181 ceph-mon[34819]: pgmap v262: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-09-21T05:45:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:00 smithi129 ceph-mon[34885]: pgmap v262: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-09-21T05:45:00.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:45:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:01.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:01 smithi028 ceph-mon[31227]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 -- 172.21.15.28:0/2871642503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970109e00 msgr2=0x7f1970110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2871642503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970109e00 0x7f1970110630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1968009ca0 tx=0x7f196802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 -- 172.21.15.28:0/2871642503 shutdown_connections 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2871642503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970109e00 0x7f1970110630 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2871642503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19701094b0 0x7f19701098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2871642503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1970103bc0 0x7f1970103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 -- 172.21.15.28:0/2871642503 >> 172.21.15.28:0/2871642503 conn(0x7f197007b910 msgr2=0x7f197007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 -- 172.21.15.28:0/2871642503 shutdown_connections 2024-09-21T05:45:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.397+0000 7f1974a6b640 1 -- 172.21.15.28:0/2871642503 wait complete. 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 Processor -- start 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 -- start start 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19701094b0 0x7f197019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1970109e00 0x7f197019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1970113ac0 con 0x7f1970103bc0 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1970113940 con 0x7f1970109e00 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f1974a6b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1970113c40 con 0x7f19701094b0 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f196e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f196e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.398+0000 7f196e59c640 1 -- 172.21.15.28:0/2474359546 learned_addr learned my addr 172.21.15.28:0/2474359546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196ed9d640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1970109e00 0x7f197019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196dd9b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19701094b0 0x7f197019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 -- 172.21.15.28:0/2474359546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19701094b0 msgr2=0x7f197019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19701094b0 0x7f197019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 -- 172.21.15.28:0/2474359546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1970109e00 msgr2=0x7f197019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1970109e00 0x7f197019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 -- 172.21.15.28:0/2474359546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19701a03c0 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f196e59c640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f196000e9f0 tx=0x7f196000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196000cd80 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19600054d0 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960010690 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19701a06b0 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.399+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19701a6e10 con 0x7f1970103bc0 2024-09-21T05:45:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.400+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1970103fd0 con 0x7f1970103bc0 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.403+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f1960004e30 con 0x7f1970103bc0 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.404+0000 7f19677fe640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 0x7f195c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.404+0000 7f196dd9b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 0x7f195c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.404+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f19600e7ba0 con 0x7f1970103bc0 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.404+0000 7f196dd9b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 0x7f195c0b8b00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1958005fd0 tx=0x7f1958005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.407+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1960056710 con 0x7f1970103bc0 2024-09-21T05:45:01.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:01 smithi181 ceph-mon[34819]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:45:01.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:01 smithi129 ceph-mon[34885]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:45:01.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.634+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f197019b580 con 0x7f1970103bc0 2024-09-21T05:45:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.635+0000 7f19677fe640 1 -- 172.21.15.28:0/2474359546 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7f19600ae290 con 0x7f1970103bc0 2024-09-21T05:45:01.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:01.637 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:45:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 msgr2=0x7f195c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 0x7f195c0b8b00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1958005fd0 tx=0x7f1958005850 comp rx=0 tx=0).stop 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 msgr2=0x7f197019a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f196000e9f0 tx=0x7f196000edf0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 shutdown_connections 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1970109e00 0x7f197019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19701094b0 0x7f197019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f195c0b65f0 0x7f195c0b8b00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 --2- 172.21.15.28:0/2474359546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1970103bc0 0x7f197019a450 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 >> 172.21.15.28:0/2474359546 conn(0x7f197007b910 msgr2=0x7f1970105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 shutdown_connections 2024-09-21T05:45:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:01.638+0000 7f1974a6b640 1 -- 172.21.15.28:0/2474359546 wait complete. 2024-09-21T05:45:02.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:02 smithi028 ceph-mon[31227]: pgmap v263: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.4 KiB/s wr, 21 op/s 2024-09-21T05:45:02.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:02 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2474359546' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:02.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:45:02.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:02 smithi181 ceph-mon[34819]: pgmap v263: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.4 KiB/s wr, 21 op/s 2024-09-21T05:45:02.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:02 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2474359546' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:02 smithi129 ceph-mon[34885]: pgmap v263: 97 pgs: 97 active+clean; 612 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.4 KiB/s wr, 21 op/s 2024-09-21T05:45:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:02 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2474359546' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 -- 172.21.15.28:0/3267422194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c108d20 msgr2=0x7f142c10f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 --2- 172.21.15.28:0/3267422194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c108d20 0x7f142c10f550 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1424009ca0 tx=0x7f142402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 -- 172.21.15.28:0/3267422194 shutdown_connections 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 --2- 172.21.15.28:0/3267422194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c108d20 0x7f142c10f550 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 --2- 172.21.15.28:0/3267422194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c1083d0 0x7f142c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 --2- 172.21.15.28:0/3267422194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c102ae0 0x7f142c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 -- 172.21.15.28:0/3267422194 >> 172.21.15.28:0/3267422194 conn(0x7f142c079760 msgr2=0x7f142c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 -- 172.21.15.28:0/3267422194 shutdown_connections 2024-09-21T05:45:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.038+0000 7f1430f6f640 1 -- 172.21.15.28:0/3267422194 wait complete. 2024-09-21T05:45:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 Processor -- start 2024-09-21T05:45:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 -- start start 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c1083d0 0x7f142c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c108d20 0x7f142c1a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f142c113aa0 con 0x7f142c102ae0 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f142c113920 con 0x7f142c1083d0 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1430f6f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f142c113c20 con 0x7f142c108d20 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f142a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f142a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59040/0 (socket says 172.21.15.28:59040) 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f142a59c640 1 -- 172.21.15.28:0/2337577069 learned_addr learned my addr 172.21.15.28:0/2337577069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:03.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.039+0000 7f1429d9b640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c1083d0 0x7f142c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142ad9d640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c108d20 0x7f142c1a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 -- 172.21.15.28:0/2337577069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c108d20 msgr2=0x7f142c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c108d20 0x7f142c1a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 -- 172.21.15.28:0/2337577069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c1083d0 msgr2=0x7f142c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c1083d0 0x7f142c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 -- 172.21.15.28:0/2337577069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142c1a2620 con 0x7f142c102ae0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f142a59c640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f141c00d950 tx=0x7f141c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c017070 con 0x7f142c102ae0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f141c009c30 con 0x7f142c102ae0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c00f620 con 0x7f142c102ae0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142c1a2910 con 0x7f142c102ae0 2024-09-21T05:45:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.040+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142c1a9070 con 0x7f142c102ae0 2024-09-21T05:45:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.041+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142c103d10 con 0x7f142c102ae0 2024-09-21T05:45:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.043+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f141c010460 con 0x7f142c102ae0 2024-09-21T05:45:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.044+0000 7f14237fe640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 0x7f14180b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.044+0000 7f1429d9b640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 0x7f14180b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.044+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f141c0e7150 con 0x7f142c102ae0 2024-09-21T05:45:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.044+0000 7f1429d9b640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 0x7f14180b8b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f142c06b1b0 tx=0x7f1414005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.048+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f141c0ad8a0 con 0x7f142c102ae0 2024-09-21T05:45:03.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.242+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f142c1a9350 con 0x7f142c102ae0 2024-09-21T05:45:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.243+0000 7f14237fe640 1 -- 172.21.15.28:0/2337577069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+22595 (secure 0 0 0) 0x7f141c0d0090 con 0x7f142c102ae0 2024-09-21T05:45:03.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:03.246 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":71,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:44:39.762020+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:45:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 msgr2=0x7f14180b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 0x7f14180b8b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f142c06b1b0 tx=0x7f1414005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 msgr2=0x7f142c19c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f141c00d950 tx=0x7f141c00de10 comp rx=0 tx=0).stop 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 shutdown_connections 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f142c108d20 0x7f142c1a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f142c1083d0 0x7f142c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f14180b65f0 0x7f14180b8b00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 --2- 172.21.15.28:0/2337577069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f142c102ae0 0x7f142c19c6b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 >> 172.21.15.28:0/2337577069 conn(0x7f142c079760 msgr2=0x7f142c104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 shutdown_connections 2024-09-21T05:45:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:03.246+0000 7f1430f6f640 1 -- 172.21.15.28:0/2337577069 wait complete. 2024-09-21T05:45:03.890 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T05:45:03.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T05:45:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:04.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:04 smithi028 ceph-mon[31227]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 26 op/s 2024-09-21T05:45:04.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:04 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2337577069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:04 smithi181 ceph-mon[34819]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 26 op/s 2024-09-21T05:45:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:04 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2337577069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 -- 172.21.15.28:0/161642244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0109df0 msgr2=0x7fc8a0110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 --2- 172.21.15.28:0/161642244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0109df0 0x7fc8a0110620 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc898009ca0 tx=0x7fc89802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 -- 172.21.15.28:0/161642244 shutdown_connections 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 --2- 172.21.15.28:0/161642244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0109df0 0x7fc8a0110620 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 --2- 172.21.15.28:0/161642244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 0x7fc8a01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 --2- 172.21.15.28:0/161642244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0103b60 0x7fc8a0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 -- 172.21.15.28:0/161642244 >> 172.21.15.28:0/161642244 conn(0x7fc8a007b850 msgr2=0x7fc8a007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 -- 172.21.15.28:0/161642244 shutdown_connections 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.589+0000 7fc8a5561640 1 -- 172.21.15.28:0/161642244 wait complete. 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.590+0000 7fc8a5561640 1 Processor -- start 2024-09-21T05:45:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 -- start start 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 0x7fc8a019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0109df0 0x7fc8a019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0113ab0 con 0x7fc8a0103b60 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0113930 con 0x7fc8a01094a0 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc8a5561640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0113c30 con 0x7fc8a0109df0 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59068/0 (socket says 172.21.15.28:59068) 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89effd640 1 -- 172.21.15.28:0/219779612 learned_addr learned my addr 172.21.15.28:0/219779612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89f7fe640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0109df0 0x7fc8a019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89e7fc640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 0x7fc8a019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89effd640 1 -- 172.21.15.28:0/219779612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0109df0 msgr2=0x7fc8a019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.591+0000 7fc89effd640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0109df0 0x7fc8a019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc89effd640 1 -- 172.21.15.28:0/219779612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 msgr2=0x7fc8a019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc89effd640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 0x7fc8a019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc89effd640 1 -- 172.21.15.28:0/219779612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a01a03d0 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc89effd640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc89000e9f0 tx=0x7fc89000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89000cd80 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8900054d0 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a01a06c0 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890010690 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.592+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a01a6e20 con 0x7fc8a0103b60 2024-09-21T05:45:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.593+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a0103f70 con 0x7fc8a0103b60 2024-09-21T05:45:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.595+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fc890004e30 con 0x7fc8a0103b60 2024-09-21T05:45:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.595+0000 7fc887fff640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 0x7fc8880b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.595+0000 7fc89e7fc640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 0x7fc8880b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.595+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7fc8900e7a90 con 0x7fc8a0103b60 2024-09-21T05:45:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.596+0000 7fc89e7fc640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 0x7fc8880b8b00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0107130 tx=0x7fc894005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.599+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc8900ae1e0 con 0x7fc8a0103b60 2024-09-21T05:45:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:04 smithi129 ceph-mon[34885]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 26 op/s 2024-09-21T05:45:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:04 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2337577069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.787+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8a0063800 con 0x7fc8880b65f0 2024-09-21T05:45:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.791+0000 7fc887fff640 1 -- 172.21.15.28:0/219779612 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258766 (secure 0 0 0) 0x7fc8a0063800 con 0x7fc8880b65f0 2024-09-21T05:45:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:04.796 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 msgr2=0x7fc8880b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 0x7fc8880b8b00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0107130 tx=0x7fc894005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 msgr2=0x7fc8a019a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc89000e9f0 tx=0x7fc89000edf0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 shutdown_connections 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a0109df0 0x7fc8a019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a01094a0 0x7fc8a019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc8880b65f0 0x7fc8880b8b00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 --2- 172.21.15.28:0/219779612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8a0103b60 0x7fc8a019a460 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 >> 172.21.15.28:0/219779612 conn(0x7fc8a007b850 msgr2=0x7fc8a01074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 shutdown_connections 2024-09-21T05:45:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:04.795+0000 7fc8a5561640 1 -- 172.21.15.28:0/219779612 wait complete. 2024-09-21T05:45:05.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-21T05:45:04.716170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651670,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":226,"num_read_kb":210,"num_write":359,"num_write_kb":2278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385416,"kb_used_data":11208,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556408,"statfs":{"total":1151940427776,"available":1151545761792,"internally_reserved":0,"allocated":11476992,"data_stored":6496906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":55296,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":131,"num_read_kb":129,"num_write":85,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003252"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133507+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:44:40.133507+0000","last_peered":"2024-09-21T05:44:40.133507+0000","last_clean":"2024-09-21T05:44:40.133507+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:44:40.133507+0000","last_undegraded":"2024-09-21T05:44:40.133507+0000","last_fullsized":"2024-09-21T05:44:40.133507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137718+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:44:40.137718+0000","last_peered":"2024-09-21T05:44:40.137718+0000","last_clean":"2024-09-21T05:44:40.137718+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:44:40.137718+0000","last_undegraded":"2024-09-21T05:44:40.137718+0000","last_fullsized":"2024-09-21T05:44:40.137718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137508+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:44:40.137508+0000","last_peered":"2024-09-21T05:44:40.137508+0000","last_clean":"2024-09-21T05:44:40.137508+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:44:40.137508+0000","last_undegraded":"2024-09-21T05:44:40.137508+0000","last_fullsized":"2024-09-21T05:44:40.137508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124984+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:44:40.124984+0000","last_peered":"2024-09-21T05:44:40.124984+0000","last_clean":"2024-09-21T05:44:40.124984+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:44:40.124984+0000","last_undegraded":"2024-09-21T05:44:40.124984+0000","last_fullsized":"2024-09-21T05:44:40.124984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136380+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:44:40.136380+0000","last_peered":"2024-09-21T05:44:40.136380+0000","last_clean":"2024-09-21T05:44:40.136380+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:44:40.136380+0000","last_undegraded":"2024-09-21T05:44:40.136380+0000","last_fullsized":"2024-09-21T05:44:40.136380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133353+0000","last_change":"2024-09-21T05:44:39.131528+0000","last_active":"2024-09-21T05:44:40.133353+0000","last_peered":"2024-09-21T05:44:40.133353+0000","last_clean":"2024-09-21T05:44:40.133353+0000","last_became_active":"2024-09-21T05:44:39.131104+0000","last_became_peered":"2024-09-21T05:44:39.131104+0000","last_unstale":"2024-09-21T05:44:40.133353+0000","last_undegraded":"2024-09-21T05:44:40.133353+0000","last_fullsized":"2024-09-21T05:44:40.133353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:22.850192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139362+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:44:40.139362+0000","last_peered":"2024-09-21T05:44:40.139362+0000","last_clean":"2024-09-21T05:44:40.139362+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:44:40.139362+0000","last_undegraded":"2024-09-21T05:44:40.139362+0000","last_fullsized":"2024-09-21T05:44:40.139362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133385+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:44:40.133385+0000","last_peered":"2024-09-21T05:44:40.133385+0000","last_clean":"2024-09-21T05:44:40.133385+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:44:40.133385+0000","last_undegraded":"2024-09-21T05:44:40.133385+0000","last_fullsized":"2024-09-21T05:44:40.133385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138206+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:44:40.138206+0000","last_peered":"2024-09-21T05:44:40.138206+0000","last_clean":"2024-09-21T05:44:40.138206+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:44:40.138206+0000","last_undegraded":"2024-09-21T05:44:40.138206+0000","last_fullsized":"2024-09-21T05:44:40.138206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134036+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:44:40.134036+0000","last_peered":"2024-09-21T05:44:40.134036+0000","last_clean":"2024-09-21T05:44:40.134036+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:44:40.134036+0000","last_undegraded":"2024-09-21T05:44:40.134036+0000","last_fullsized":"2024-09-21T05:44:40.134036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139995+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:44:40.139995+0000","last_peered":"2024-09-21T05:44:40.139995+0000","last_clean":"2024-09-21T05:44:40.139995+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:44:40.139995+0000","last_undegraded":"2024-09-21T05:44:40.139995+0000","last_fullsized":"2024-09-21T05:44:40.139995+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125119+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:44:40.125119+0000","last_peered":"2024-09-21T05:44:40.125119+0000","last_clean":"2024-09-21T05:44:40.125119+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:44:40.125119+0000","last_undegraded":"2024-09-21T05:44:40.125119+0000","last_fullsized":"2024-09-21T05:44:40.125119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124242+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:44:40.124242+0000","last_peered":"2024-09-21T05:44:40.124242+0000","last_clean":"2024-09-21T05:44:40.124242+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:44:40.124242+0000","last_undegraded":"2024-09-21T05:44:40.124242+0000","last_fullsized":"2024-09-21T05:44:40.124242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137028+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:44:40.137028+0000","last_peered":"2024-09-21T05:44:40.137028+0000","last_clean":"2024-09-21T05:44:40.137028+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:44:40.137028+0000","last_undegraded":"2024-09-21T05:44:40.137028+0000","last_fullsized":"2024-09-21T05:44:40.137028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136835+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:44:40.136835+0000","last_peered":"2024-09-21T05:44:40.136835+0000","last_clean":"2024-09-21T05:44:40.136835+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:44:40.136835+0000","last_undegraded":"2024-09-21T05:44:40.136835+0000","last_fullsized":"2024-09-21T05:44:40.136835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136326+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:44:40.136326+0000","last_peered":"2024-09-21T05:44:40.136326+0000","last_clean":"2024-09-21T05:44:40.136326+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:44:40.136326+0000","last_undegraded":"2024-09-21T05:44:40.136326+0000","last_fullsized":"2024-09-21T05:44:40.136326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133696+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:44:40.133696+0000","last_peered":"2024-09-21T05:44:40.133696+0000","last_clean":"2024-09-21T05:44:40.133696+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:44:40.133696+0000","last_undegraded":"2024-09-21T05:44:40.133696+0000","last_fullsized":"2024-09-21T05:44:40.133696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135812+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:44:40.135812+0000","last_peered":"2024-09-21T05:44:40.135812+0000","last_clean":"2024-09-21T05:44:40.135812+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:44:40.135812+0000","last_undegraded":"2024-09-21T05:44:40.135812+0000","last_fullsized":"2024-09-21T05:44:40.135812+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.126209+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:44:40.126209+0000","last_peered":"2024-09-21T05:44:40.126209+0000","last_clean":"2024-09-21T05:44:40.126209+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:44:40.126209+0000","last_undegraded":"2024-09-21T05:44:40.126209+0000","last_fullsized":"2024-09-21T05:44:40.126209+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133450+0000","last_change":"2024-09-21T05:44:39.119824+0000","last_active":"2024-09-21T05:44:40.133450+0000","last_peered":"2024-09-21T05:44:40.133450+0000","last_clean":"2024-09-21T05:44:40.133450+0000","last_became_active":"2024-09-21T05:44:39.119475+0000","last_became_peered":"2024-09-21T05:44:39.119475+0000","last_unstale":"2024-09-21T05:44:40.133450+0000","last_undegraded":"2024-09-21T05:44:40.133450+0000","last_fullsized":"2024-09-21T05:44:40.133450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:29.773122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124197+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:44:40.124197+0000","last_peered":"2024-09-21T05:44:40.124197+0000","last_clean":"2024-09-21T05:44:40.124197+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:44:40.124197+0000","last_undegraded":"2024-09-21T05:44:40.124197+0000","last_fullsized":"2024-09-21T05:44:40.124197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137091+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:44:40.137091+0000","last_peered":"2024-09-21T05:44:40.137091+0000","last_clean":"2024-09-21T05:44:40.137091+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:44:40.137091+0000","last_undegraded":"2024-09-21T05:44:40.137091+0000","last_fullsized":"2024-09-21T05:44:40.137091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133517+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:44:40.133517+0000","last_peered":"2024-09-21T05:44:40.133517+0000","last_clean":"2024-09-21T05:44:40.133517+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:44:40.133517+0000","last_undegraded":"2024-09-21T05:44:40.133517+0000","last_fullsized":"2024-09-21T05:44:40.133517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138681+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:44:40.138681+0000","last_peered":"2024-09-21T05:44:40.138681+0000","last_clean":"2024-09-21T05:44:40.138681+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:44:40.138681+0000","last_undegraded":"2024-09-21T05:44:40.138681+0000","last_fullsized":"2024-09-21T05:44:40.138681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134110+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:44:40.134110+0000","last_peered":"2024-09-21T05:44:40.134110+0000","last_clean":"2024-09-21T05:44:40.134110+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:44:40.134110+0000","last_undegraded":"2024-09-21T05:44:40.134110+0000","last_fullsized":"2024-09-21T05:44:40.134110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.126101+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:44:40.126101+0000","last_peered":"2024-09-21T05:44:40.126101+0000","last_clean":"2024-09-21T05:44:40.126101+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:44:40.126101+0000","last_undegraded":"2024-09-21T05:44:40.126101+0000","last_fullsized":"2024-09-21T05:44:40.126101+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133187+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:44:40.133187+0000","last_peered":"2024-09-21T05:44:40.133187+0000","last_clean":"2024-09-21T05:44:40.133187+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:44:40.133187+0000","last_undegraded":"2024-09-21T05:44:40.133187+0000","last_fullsized":"2024-09-21T05:44:40.133187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124134+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:44:40.124134+0000","last_peered":"2024-09-21T05:44:40.124134+0000","last_clean":"2024-09-21T05:44:40.124134+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:44:40.124134+0000","last_undegraded":"2024-09-21T05:44:40.124134+0000","last_fullsized":"2024-09-21T05:44:40.124134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137199+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:44:40.137199+0000","last_peered":"2024-09-21T05:44:40.137199+0000","last_clean":"2024-09-21T05:44:40.137199+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:44:40.137199+0000","last_undegraded":"2024-09-21T05:44:40.137199+0000","last_fullsized":"2024-09-21T05:44:40.137199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125218+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:44:40.125218+0000","last_peered":"2024-09-21T05:44:40.125218+0000","last_clean":"2024-09-21T05:44:40.125218+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:44:40.125218+0000","last_undegraded":"2024-09-21T05:44:40.125218+0000","last_fullsized":"2024-09-21T05:44:40.125218+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125790+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:44:40.125790+0000","last_peered":"2024-09-21T05:44:40.125790+0000","last_clean":"2024-09-21T05:44:40.125790+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:44:40.125790+0000","last_undegraded":"2024-09-21T05:44:40.125790+0000","last_fullsized":"2024-09-21T05:44:40.125790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.132870+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:44:40.132870+0000","last_peered":"2024-09-21T05:44:40.132870+0000","last_clean":"2024-09-21T05:44:40.132870+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:44:40.132870+0000","last_undegraded":"2024-09-21T05:44:40.132870+0000","last_fullsized":"2024-09-21T05:44:40.132870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136508+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:44:40.136508+0000","last_peered":"2024-09-21T05:44:40.136508+0000","last_clean":"2024-09-21T05:44:40.136508+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:44:40.136508+0000","last_undegraded":"2024-09-21T05:44:40.136508+0000","last_fullsized":"2024-09-21T05:44:40.136508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:01.073683+0000","last_change":"2024-09-21T05:44:37.112922+0000","last_active":"2024-09-21T05:45:01.073683+0000","last_peered":"2024-09-21T05:45:01.073683+0000","last_clean":"2024-09-21T05:45:01.073683+0000","last_became_active":"2024-09-21T05:44:37.112457+0000","last_became_peered":"2024-09-21T05:44:37.112457+0000","last_unstale":"2024-09-21T05:45:01.073683+0000","last_undegraded":"2024-09-21T05:45:01.073683+0000","last_fullsized":"2024-09-21T05:45:01.073683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:41.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139856+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:44:40.139856+0000","last_peered":"2024-09-21T05:44:40.139856+0000","last_clean":"2024-09-21T05:44:40.139856+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:44:40.139856+0000","last_undegraded":"2024-09-21T05:44:40.139856+0000","last_fullsized":"2024-09-21T05:44:40.139856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":26,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.198895+0000","last_change":"2024-09-21T05:44:37.112622+0000","last_active":"2024-09-21T05:44:50.198895+0000","last_peered":"2024-09-21T05:44:50.198895+0000","last_clean":"2024-09-21T05:44:50.198895+0000","last_became_active":"2024-09-21T05:44:37.112215+0000","last_became_peered":"2024-09-21T05:44:37.112215+0000","last_unstale":"2024-09-21T05:44:50.198895+0000","last_undegraded":"2024-09-21T05:44:50.198895+0000","last_fullsized":"2024-09-21T05:44:50.198895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:48.725835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:49.189856+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:44:49.189856+0000","last_peered":"2024-09-21T05:44:49.189856+0000","last_clean":"2024-09-21T05:44:49.189856+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:44:49.189856+0000","last_undegraded":"2024-09-21T05:44:49.189856+0000","last_fullsized":"2024-09-21T05:44:49.189856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133641+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:44:40.133641+0000","last_peered":"2024-09-21T05:44:40.133641+0000","last_clean":"2024-09-21T05:44:40.133641+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:44:40.133641+0000","last_undegraded":"2024-09-21T05:44:40.133641+0000","last_fullsized":"2024-09-21T05:44:40.133641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212702+0000","last_change":"2024-09-21T05:44:37.110808+0000","last_active":"2024-09-21T05:44:51.212702+0000","last_peered":"2024-09-21T05:44:51.212702+0000","last_clean":"2024-09-21T05:44:51.212702+0000","last_became_active":"2024-09-21T05:44:37.110376+0000","last_became_peered":"2024-09-21T05:44:37.110376+0000","last_unstale":"2024-09-21T05:44:51.212702+0000","last_undegraded":"2024-09-21T05:44:51.212702+0000","last_fullsized":"2024-09-21T05:44:51.212702+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:11.562980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135325+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:44:40.135325+0000","last_peered":"2024-09-21T05:44:40.135325+0000","last_clean":"2024-09-21T05:44:40.135325+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:44:40.135325+0000","last_undegraded":"2024-09-21T05:44:40.135325+0000","last_fullsized":"2024-09-21T05:44:40.135325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"71'4","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.074240+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:45:02.074240+0000","last_peered":"2024-09-21T05:45:02.074240+0000","last_clean":"2024-09-21T05:45:02.074240+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:45:02.074240+0000","last_undegraded":"2024-09-21T05:45:02.074240+0000","last_fullsized":"2024-09-21T05:45:02.074240+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136852+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:44:40.136852+0000","last_peered":"2024-09-21T05:44:40.136852+0000","last_clean":"2024-09-21T05:44:40.136852+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:44:40.136852+0000","last_undegraded":"2024-09-21T05:44:40.136852+0000","last_fullsized":"2024-09-21T05:44:40.136852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212070+0000","last_change":"2024-09-21T05:44:37.110361+0000","last_active":"2024-09-21T05:44:51.212070+0000","last_peered":"2024-09-21T05:44:51.212070+0000","last_clean":"2024-09-21T05:44:51.212070+0000","last_became_active":"2024-09-21T05:44:37.109912+0000","last_became_peered":"2024-09-21T05:44:37.109912+0000","last_unstale":"2024-09-21T05:44:51.212070+0000","last_undegraded":"2024-09-21T05:44:51.212070+0000","last_fullsized":"2024-09-21T05:44:51.212070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:05:00.470854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137738+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:44:40.137738+0000","last_peered":"2024-09-21T05:44:40.137738+0000","last_clean":"2024-09-21T05:44:40.137738+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:44:40.137738+0000","last_undegraded":"2024-09-21T05:44:40.137738+0000","last_fullsized":"2024-09-21T05:44:40.137738+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.197688+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:44:50.197688+0000","last_peered":"2024-09-21T05:44:50.197688+0000","last_clean":"2024-09-21T05:44:50.197688+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:44:50.197688+0000","last_undegraded":"2024-09-21T05:44:50.197688+0000","last_fullsized":"2024-09-21T05:44:50.197688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137590+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:44:40.137590+0000","last_peered":"2024-09-21T05:44:40.137590+0000","last_clean":"2024-09-21T05:44:40.137590+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:44:40.137590+0000","last_undegraded":"2024-09-21T05:44:40.137590+0000","last_fullsized":"2024-09-21T05:44:40.137590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"71'27","reported_seq":74,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.077328+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:45:02.077328+0000","last_peered":"2024-09-21T05:45:02.077328+0000","last_clean":"2024-09-21T05:45:02.077328+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:45:02.077328+0000","last_undegraded":"2024-09-21T05:45:02.077328+0000","last_fullsized":"2024-09-21T05:45:02.077328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54255,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":66,"num_write":29,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137071+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:44:40.137071+0000","last_peered":"2024-09-21T05:44:40.137071+0000","last_clean":"2024-09-21T05:44:40.137071+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:44:40.137071+0000","last_undegraded":"2024-09-21T05:44:40.137071+0000","last_fullsized":"2024-09-21T05:44:40.137071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056148+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:44:52.056148+0000","last_peered":"2024-09-21T05:44:52.056148+0000","last_clean":"2024-09-21T05:44:52.056148+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:44:52.056148+0000","last_undegraded":"2024-09-21T05:44:52.056148+0000","last_fullsized":"2024-09-21T05:44:52.056148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124014+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:44:40.124014+0000","last_peered":"2024-09-21T05:44:40.124014+0000","last_clean":"2024-09-21T05:44:40.124014+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:44:40.124014+0000","last_undegraded":"2024-09-21T05:44:40.124014+0000","last_fullsized":"2024-09-21T05:44:40.124014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212863+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:44:51.212863+0000","last_peered":"2024-09-21T05:44:51.212863+0000","last_clean":"2024-09-21T05:44:51.212863+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:44:51.212863+0000","last_undegraded":"2024-09-21T05:44:51.212863+0000","last_fullsized":"2024-09-21T05:44:51.212863+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139564+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:44:40.139564+0000","last_peered":"2024-09-21T05:44:40.139564+0000","last_clean":"2024-09-21T05:44:40.139564+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:44:40.139564+0000","last_undegraded":"2024-09-21T05:44:40.139564+0000","last_fullsized":"2024-09-21T05:44:40.139564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.074559+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:45:02.074559+0000","last_peered":"2024-09-21T05:45:02.074559+0000","last_clean":"2024-09-21T05:45:02.074559+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:45:02.074559+0000","last_undegraded":"2024-09-21T05:45:02.074559+0000","last_fullsized":"2024-09-21T05:45:02.074559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139228+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:44:40.139228+0000","last_peered":"2024-09-21T05:44:40.139228+0000","last_clean":"2024-09-21T05:44:40.139228+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:44:40.139228+0000","last_undegraded":"2024-09-21T05:44:40.139228+0000","last_fullsized":"2024-09-21T05:44:40.139228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210953+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:44:51.210953+0000","last_peered":"2024-09-21T05:44:51.210953+0000","last_clean":"2024-09-21T05:44:51.210953+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:44:51.210953+0000","last_undegraded":"2024-09-21T05:44:51.210953+0000","last_fullsized":"2024-09-21T05:44:51.210953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137685+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:44:40.137685+0000","last_peered":"2024-09-21T05:44:40.137685+0000","last_clean":"2024-09-21T05:44:40.137685+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:44:40.137685+0000","last_undegraded":"2024-09-21T05:44:40.137685+0000","last_fullsized":"2024-09-21T05:44:40.137685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.184984+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:44:51.184984+0000","last_peered":"2024-09-21T05:44:51.184984+0000","last_clean":"2024-09-21T05:44:51.184984+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:44:51.184984+0000","last_undegraded":"2024-09-21T05:44:51.184984+0000","last_fullsized":"2024-09-21T05:44:51.184984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133386+0000","last_change":"2024-09-21T05:44:39.114163+0000","last_active":"2024-09-21T05:44:40.133386+0000","last_peered":"2024-09-21T05:44:40.133386+0000","last_clean":"2024-09-21T05:44:40.133386+0000","last_became_active":"2024-09-21T05:44:39.113799+0000","last_became_peered":"2024-09-21T05:44:39.113799+0000","last_unstale":"2024-09-21T05:44:40.133386+0000","last_undegraded":"2024-09-21T05:44:40.133386+0000","last_fullsized":"2024-09-21T05:44:40.133386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.286926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211416+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:44:51.211416+0000","last_peered":"2024-09-21T05:44:51.211416+0000","last_clean":"2024-09-21T05:44:51.211416+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:44:51.211416+0000","last_undegraded":"2024-09-21T05:44:51.211416+0000","last_fullsized":"2024-09-21T05:44:51.211416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139903+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:44:40.139903+0000","last_peered":"2024-09-21T05:44:40.139903+0000","last_clean":"2024-09-21T05:44:40.139903+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:44:40.139903+0000","last_undegraded":"2024-09-21T05:44:40.139903+0000","last_fullsized":"2024-09-21T05:44:40.139903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210590+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:44:51.210590+0000","last_peered":"2024-09-21T05:44:51.210590+0000","last_clean":"2024-09-21T05:44:51.210590+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:44:51.210590+0000","last_undegraded":"2024-09-21T05:44:51.210590+0000","last_fullsized":"2024-09-21T05:44:51.210590+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.927251+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:44:40.927251+0000","last_peered":"2024-09-21T05:44:40.927251+0000","last_clean":"2024-09-21T05:44:40.927251+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:44:40.927251+0000","last_undegraded":"2024-09-21T05:44:40.927251+0000","last_fullsized":"2024-09-21T05:44:40.927251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"71'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:00.072179+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:45:00.072179+0000","last_peered":"2024-09-21T05:45:00.072179+0000","last_clean":"2024-09-21T05:45:00.072179+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:45:00.072179+0000","last_undegraded":"2024-09-21T05:45:00.072179+0000","last_fullsized":"2024-09-21T05:45:00.072179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.140027+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:44:40.140027+0000","last_peered":"2024-09-21T05:44:40.140027+0000","last_clean":"2024-09-21T05:44:40.140027+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:44:40.140027+0000","last_undegraded":"2024-09-21T05:44:40.140027+0000","last_fullsized":"2024-09-21T05:44:40.140027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":88,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139323+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:44:40.139323+0000","last_peered":"2024-09-21T05:44:40.139323+0000","last_clean":"2024-09-21T05:44:40.139323+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:44:40.139323+0000","last_undegraded":"2024-09-21T05:44:40.139323+0000","last_fullsized":"2024-09-21T05:44:40.139323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":28,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056608+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:44:52.056608+0000","last_peered":"2024-09-21T05:44:52.056608+0000","last_clean":"2024-09-21T05:44:52.056608+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:44:52.056608+0000","last_undegraded":"2024-09-21T05:44:52.056608+0000","last_fullsized":"2024-09-21T05:44:52.056608+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135011+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:44:40.135011+0000","last_peered":"2024-09-21T05:44:40.135011+0000","last_clean":"2024-09-21T05:44:40.135011+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:44:40.135011+0000","last_undegraded":"2024-09-21T05:44:40.135011+0000","last_fullsized":"2024-09-21T05:44:40.135011+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:49.190605+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:44:49.190605+0000","last_peered":"2024-09-21T05:44:49.190605+0000","last_clean":"2024-09-21T05:44:49.190605+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:44:49.190605+0000","last_undegraded":"2024-09-21T05:44:49.190605+0000","last_fullsized":"2024-09-21T05:44:49.190605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124412+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:44:40.124412+0000","last_peered":"2024-09-21T05:44:40.124412+0000","last_clean":"2024-09-21T05:44:40.124412+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:44:40.124412+0000","last_undegraded":"2024-09-21T05:44:40.124412+0000","last_fullsized":"2024-09-21T05:44:40.124412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:00.071323+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:45:00.071323+0000","last_peered":"2024-09-21T05:45:00.071323+0000","last_clean":"2024-09-21T05:45:00.071323+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:45:00.071323+0000","last_undegraded":"2024-09-21T05:45:00.071323+0000","last_fullsized":"2024-09-21T05:45:00.071323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138932+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:44:40.138932+0000","last_peered":"2024-09-21T05:44:40.138932+0000","last_clean":"2024-09-21T05:44:40.138932+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:44:40.138932+0000","last_undegraded":"2024-09-21T05:44:40.138932+0000","last_fullsized":"2024-09-21T05:44:40.138932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:59.070068+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:44:59.070068+0000","last_peered":"2024-09-21T05:44:59.070068+0000","last_clean":"2024-09-21T05:44:59.070068+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:44:59.070068+0000","last_undegraded":"2024-09-21T05:44:59.070068+0000","last_fullsized":"2024-09-21T05:44:59.070068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133684+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:44:40.133684+0000","last_peered":"2024-09-21T05:44:40.133684+0000","last_clean":"2024-09-21T05:44:40.133684+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:44:40.133684+0000","last_undegraded":"2024-09-21T05:44:40.133684+0000","last_fullsized":"2024-09-21T05:44:40.133684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136454+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:44:40.136454+0000","last_peered":"2024-09-21T05:44:40.136454+0000","last_clean":"2024-09-21T05:44:40.136454+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:44:40.136454+0000","last_undegraded":"2024-09-21T05:44:40.136454+0000","last_fullsized":"2024-09-21T05:44:40.136454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:42.530528+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:44:42.530528+0000","last_peered":"2024-09-21T05:44:42.530528+0000","last_clean":"2024-09-21T05:44:42.530528+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:44:42.530528+0000","last_undegraded":"2024-09-21T05:44:42.530528+0000","last_fullsized":"2024-09-21T05:44:42.530528+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211397+0000","last_change":"2024-09-21T05:44:37.270537+0000","last_active":"2024-09-21T05:44:51.211397+0000","last_peered":"2024-09-21T05:44:51.211397+0000","last_clean":"2024-09-21T05:44:51.211397+0000","last_became_active":"2024-09-21T05:44:37.269845+0000","last_became_peered":"2024-09-21T05:44:37.269845+0000","last_unstale":"2024-09-21T05:44:51.211397+0000","last_undegraded":"2024-09-21T05:44:51.211397+0000","last_fullsized":"2024-09-21T05:44:51.211397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:39.402650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124501+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:44:40.124501+0000","last_peered":"2024-09-21T05:44:40.124501+0000","last_clean":"2024-09-21T05:44:40.124501+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:44:40.124501+0000","last_undegraded":"2024-09-21T05:44:40.124501+0000","last_fullsized":"2024-09-21T05:44:40.124501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"71'12","reported_seq":31,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:01.131922+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:01.131922+0000","last_peered":"2024-09-21T05:45:01.131922+0000","last_clean":"2024-09-21T05:45:01.131922+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:01.131922+0000","last_undegraded":"2024-09-21T05:45:01.131922+0000","last_fullsized":"2024-09-21T05:45:01.131922+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134274+0000","last_change":"2024-09-21T05:44:39.125685+0000","last_active":"2024-09-21T05:44:40.134274+0000","last_peered":"2024-09-21T05:44:40.134274+0000","last_clean":"2024-09-21T05:44:40.134274+0000","last_became_active":"2024-09-21T05:44:39.125308+0000","last_became_peered":"2024-09-21T05:44:39.125308+0000","last_unstale":"2024-09-21T05:44:40.134274+0000","last_undegraded":"2024-09-21T05:44:40.134274+0000","last_fullsized":"2024-09-21T05:44:40.134274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:48.599727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211139+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:44:51.211139+0000","last_peered":"2024-09-21T05:44:51.211139+0000","last_clean":"2024-09-21T05:44:51.211139+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:44:51.211139+0000","last_undegraded":"2024-09-21T05:44:51.211139+0000","last_fullsized":"2024-09-21T05:44:51.211139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137636+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:44:40.137636+0000","last_peered":"2024-09-21T05:44:40.137636+0000","last_clean":"2024-09-21T05:44:40.137636+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:44:40.137636+0000","last_undegraded":"2024-09-21T05:44:40.137636+0000","last_fullsized":"2024-09-21T05:44:40.137636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:48.180015+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:44:48.180015+0000","last_peered":"2024-09-21T05:44:48.180015+0000","last_clean":"2024-09-21T05:44:48.180015+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:44:48.180015+0000","last_undegraded":"2024-09-21T05:44:48.180015+0000","last_fullsized":"2024-09-21T05:44:48.180015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136233+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:44:40.136233+0000","last_peered":"2024-09-21T05:44:40.136233+0000","last_clean":"2024-09-21T05:44:40.136233+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:44:40.136233+0000","last_undegraded":"2024-09-21T05:44:40.136233+0000","last_fullsized":"2024-09-21T05:44:40.136233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":26,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210212+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:44:51.210212+0000","last_peered":"2024-09-21T05:44:51.210212+0000","last_clean":"2024-09-21T05:44:51.210212+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:44:51.210212+0000","last_undegraded":"2024-09-21T05:44:51.210212+0000","last_fullsized":"2024-09-21T05:44:51.210212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.132996+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:44:40.132996+0000","last_peered":"2024-09-21T05:44:40.132996+0000","last_clean":"2024-09-21T05:44:40.132996+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:44:40.132996+0000","last_undegraded":"2024-09-21T05:44:40.132996+0000","last_fullsized":"2024-09-21T05:44:40.132996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.198543+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:44:50.198543+0000","last_peered":"2024-09-21T05:44:50.198543+0000","last_clean":"2024-09-21T05:44:50.198543+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:44:50.198543+0000","last_undegraded":"2024-09-21T05:44:50.198543+0000","last_fullsized":"2024-09-21T05:44:50.198543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134191+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:44:40.134191+0000","last_peered":"2024-09-21T05:44:40.134191+0000","last_clean":"2024-09-21T05:44:40.134191+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:44:40.134191+0000","last_undegraded":"2024-09-21T05:44:40.134191+0000","last_fullsized":"2024-09-21T05:44:40.134191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:48.180657+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:44:48.180657+0000","last_peered":"2024-09-21T05:44:48.180657+0000","last_clean":"2024-09-21T05:44:48.180657+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:44:48.180657+0000","last_undegraded":"2024-09-21T05:44:48.180657+0000","last_fullsized":"2024-09-21T05:44:48.180657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139404+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:44:40.139404+0000","last_peered":"2024-09-21T05:44:40.139404+0000","last_clean":"2024-09-21T05:44:40.139404+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:44:40.139404+0000","last_undegraded":"2024-09-21T05:44:40.139404+0000","last_fullsized":"2024-09-21T05:44:40.139404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139111+0000","last_change":"2024-09-21T05:44:39.127591+0000","last_active":"2024-09-21T05:44:40.139111+0000","last_peered":"2024-09-21T05:44:40.139111+0000","last_clean":"2024-09-21T05:44:40.139111+0000","last_became_active":"2024-09-21T05:44:39.127179+0000","last_became_peered":"2024-09-21T05:44:39.127179+0000","last_unstale":"2024-09-21T05:44:40.139111+0000","last_undegraded":"2024-09-21T05:44:40.139111+0000","last_fullsized":"2024-09-21T05:44:40.139111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:55.173951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210759+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:44:51.210759+0000","last_peered":"2024-09-21T05:44:51.210759+0000","last_clean":"2024-09-21T05:44:51.210759+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:44:51.210759+0000","last_undegraded":"2024-09-21T05:44:51.210759+0000","last_fullsized":"2024-09-21T05:44:51.210759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136466+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:44:40.136466+0000","last_peered":"2024-09-21T05:44:40.136466+0000","last_clean":"2024-09-21T05:44:40.136466+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:44:40.136466+0000","last_undegraded":"2024-09-21T05:44:40.136466+0000","last_fullsized":"2024-09-21T05:44:40.136466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056120+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:44:52.056120+0000","last_peered":"2024-09-21T05:44:52.056120+0000","last_clean":"2024-09-21T05:44:52.056120+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:44:52.056120+0000","last_undegraded":"2024-09-21T05:44:52.056120+0000","last_fullsized":"2024-09-21T05:44:52.056120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137929+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:44:40.137929+0000","last_peered":"2024-09-21T05:44:40.137929+0000","last_clean":"2024-09-21T05:44:40.137929+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:44:40.137929+0000","last_undegraded":"2024-09-21T05:44:40.137929+0000","last_fullsized":"2024-09-21T05:44:40.137929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.140161+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:44:40.140161+0000","last_peered":"2024-09-21T05:44:40.140161+0000","last_clean":"2024-09-21T05:44:40.140161+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:44:40.140161+0000","last_undegraded":"2024-09-21T05:44:40.140161+0000","last_fullsized":"2024-09-21T05:44:40.140161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134024+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:44:40.134024+0000","last_peered":"2024-09-21T05:44:40.134024+0000","last_clean":"2024-09-21T05:44:40.134024+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:44:40.134024+0000","last_undegraded":"2024-09-21T05:44:40.134024+0000","last_fullsized":"2024-09-21T05:44:40.134024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.073490+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:45:02.073490+0000","last_peered":"2024-09-21T05:45:02.073490+0000","last_clean":"2024-09-21T05:45:02.073490+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:02.073490+0000","last_undegraded":"2024-09-21T05:45:02.073490+0000","last_fullsized":"2024-09-21T05:45:02.073490+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61068,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":145,"num_read_kb":141,"num_write":234,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":183204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874258,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36100,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709052,"statfs":{"total":95995035648,"available":95958069248,"internally_reserved":0,"allocated":774144,"data_stored":381583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037782,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712532,"statfs":{"total":95995035648,"available":95961632768,"internally_reserved":0,"allocated":1470464,"data_stored":1026476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201307,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36124,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709028,"statfs":{"total":95995035648,"available":95958044672,"internally_reserved":0,"allocated":798720,"data_stored":381122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":794624,"data_stored":381500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561060,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708448,"statfs":{"total":95995035648,"available":95957450752,"internally_reserved":0,"allocated":1392640,"data_stored":973627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724585,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":782336,"data_stored":380839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888110,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1404928,"data_stored":973305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":782336,"data_stored":380700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":798720,"data_stored":380664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":847872,"data_stored":427566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542210,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":827392,"data_stored":426955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":802816,"data_stored":382569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":55991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:45:05.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T05:45:05.736 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- 172.21.15.28:0/3168536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102a80 msgr2=0x7fd2fc102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 --2- 172.21.15.28:0/3168536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102a80 0x7fd2fc102e90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009900 tx=0x7fd2f802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- 172.21.15.28:0/3168536368 shutdown_connections 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 --2- 172.21.15.28:0/3168536368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc108d10 0x7fd2fc10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 --2- 172.21.15.28:0/3168536368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 --2- 172.21.15.28:0/3168536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102a80 0x7fd2fc102e90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- 172.21.15.28:0/3168536368 >> 172.21.15.28:0/3168536368 conn(0x7fd2fc079740 msgr2=0x7fd2fc079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- 172.21.15.28:0/3168536368 shutdown_connections 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- 172.21.15.28:0/3168536368 wait complete. 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 Processor -- start 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.195+0000 7fd303d65640 1 -- start start 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc102a80 0x7fd2fc199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc108d10 0x7fd2fc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc112980 con 0x7fd2fc1083c0 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc112800 con 0x7fd2fc102a80 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd303d65640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc112b00 con 0x7fd2fc108d10 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54490/0 (socket says 172.21.15.28:54490) 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 -- 172.21.15.28:0/3629199528 learned_addr learned my addr 172.21.15.28:0/3629199528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd302302640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc108d10 0x7fd2fc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301b01640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc102a80 0x7fd2fc199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 -- 172.21.15.28:0/3629199528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc108d10 msgr2=0x7fd2fc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc108d10 0x7fd2fc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 -- 172.21.15.28:0/3629199528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc102a80 msgr2=0x7fd2fc199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc102a80 0x7fd2fc199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 -- 172.21.15.28:0/3629199528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc19f2c0 con 0x7fd2fc1083c0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.196+0000 7fd301300640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00d770 tx=0x7fd2ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.197+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec017070 con 0x7fd2fc1083c0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.197+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ec009c30 con 0x7fd2fc1083c0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.197+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc19f5b0 con 0x7fd2fc1083c0 2024-09-21T05:45:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.197+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00f620 con 0x7fd2fc1083c0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.197+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc1a5d10 con 0x7fd2fc1083c0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.198+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc103d00 con 0x7fd2fc1083c0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.200+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fd2ec01a070 con 0x7fd2fc1083c0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.201+0000 7fd2f2ffd640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 0x7fd2e80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.201+0000 7fd301b01640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 0x7fd2e80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.201+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7fd2ec0e6fe0 con 0x7fd2fc1083c0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.202+0000 7fd301b01640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 0x7fd2e80b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8031440 tx=0x7fd2f8005d50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.204+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd2ec009de0 con 0x7fd2fc1083c0 2024-09-21T05:45:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.390+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2fc063800 con 0x7fd2e80b65f0 2024-09-21T05:45:06.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:06 smithi028 ceph-mon[31227]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 17 op/s 2024-09-21T05:45:06.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:06 smithi028 ceph-mon[31227]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:06.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.395+0000 7fd2f2ffd640 1 -- 172.21.15.28:0/3629199528 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258766 (secure 0 0 0) 0x7fd2fc063800 con 0x7fd2e80b65f0 2024-09-21T05:45:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:45:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 msgr2=0x7fd2e80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 0x7fd2e80b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8031440 tx=0x7fd2f8005d50 comp rx=0 tx=0).stop 2024-09-21T05:45:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 msgr2=0x7fd2fc1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00d770 tx=0x7fd2ec00dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 shutdown_connections 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2fc108d10 0x7fd2fc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd2e80b65f0 0x7fd2e80b8b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc1083c0 0x7fd2fc1998f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 --2- 172.21.15.28:0/3629199528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2fc102a80 0x7fd2fc199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 >> 172.21.15.28:0/3629199528 conn(0x7fd2fc079740 msgr2=0x7fd2fc101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 shutdown_connections 2024-09-21T05:45:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:06.399+0000 7fd303d65640 1 -- 172.21.15.28:0/3629199528 wait complete. 2024-09-21T05:45:06.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:06 smithi181 ceph-mon[34819]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 17 op/s 2024-09-21T05:45:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:06 smithi181 ceph-mon[34819]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:06 smithi129 ceph-mon[34885]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 17 op/s 2024-09-21T05:45:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:06 smithi129 ceph-mon[34885]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:07.180 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-21T05:45:04.716170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651670,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":226,"num_read_kb":210,"num_write":359,"num_write_kb":2278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385416,"kb_used_data":11208,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556408,"statfs":{"total":1151940427776,"available":1151545761792,"internally_reserved":0,"allocated":11476992,"data_stored":6496906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":55296,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":131,"num_read_kb":129,"num_write":85,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003252"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133507+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:44:40.133507+0000","last_peered":"2024-09-21T05:44:40.133507+0000","last_clean":"2024-09-21T05:44:40.133507+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:44:40.133507+0000","last_undegraded":"2024-09-21T05:44:40.133507+0000","last_fullsized":"2024-09-21T05:44:40.133507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137718+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:44:40.137718+0000","last_peered":"2024-09-21T05:44:40.137718+0000","last_clean":"2024-09-21T05:44:40.137718+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:44:40.137718+0000","last_undegraded":"2024-09-21T05:44:40.137718+0000","last_fullsized":"2024-09-21T05:44:40.137718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137508+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:44:40.137508+0000","last_peered":"2024-09-21T05:44:40.137508+0000","last_clean":"2024-09-21T05:44:40.137508+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:44:40.137508+0000","last_undegraded":"2024-09-21T05:44:40.137508+0000","last_fullsized":"2024-09-21T05:44:40.137508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124984+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:44:40.124984+0000","last_peered":"2024-09-21T05:44:40.124984+0000","last_clean":"2024-09-21T05:44:40.124984+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:44:40.124984+0000","last_undegraded":"2024-09-21T05:44:40.124984+0000","last_fullsized":"2024-09-21T05:44:40.124984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136380+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:44:40.136380+0000","last_peered":"2024-09-21T05:44:40.136380+0000","last_clean":"2024-09-21T05:44:40.136380+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:44:40.136380+0000","last_undegraded":"2024-09-21T05:44:40.136380+0000","last_fullsized":"2024-09-21T05:44:40.136380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133353+0000","last_change":"2024-09-21T05:44:39.131528+0000","last_active":"2024-09-21T05:44:40.133353+0000","last_peered":"2024-09-21T05:44:40.133353+0000","last_clean":"2024-09-21T05:44:40.133353+0000","last_became_active":"2024-09-21T05:44:39.131104+0000","last_became_peered":"2024-09-21T05:44:39.131104+0000","last_unstale":"2024-09-21T05:44:40.133353+0000","last_undegraded":"2024-09-21T05:44:40.133353+0000","last_fullsized":"2024-09-21T05:44:40.133353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:22.850192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139362+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:44:40.139362+0000","last_peered":"2024-09-21T05:44:40.139362+0000","last_clean":"2024-09-21T05:44:40.139362+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:44:40.139362+0000","last_undegraded":"2024-09-21T05:44:40.139362+0000","last_fullsized":"2024-09-21T05:44:40.139362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133385+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:44:40.133385+0000","last_peered":"2024-09-21T05:44:40.133385+0000","last_clean":"2024-09-21T05:44:40.133385+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:44:40.133385+0000","last_undegraded":"2024-09-21T05:44:40.133385+0000","last_fullsized":"2024-09-21T05:44:40.133385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138206+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:44:40.138206+0000","last_peered":"2024-09-21T05:44:40.138206+0000","last_clean":"2024-09-21T05:44:40.138206+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:44:40.138206+0000","last_undegraded":"2024-09-21T05:44:40.138206+0000","last_fullsized":"2024-09-21T05:44:40.138206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134036+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:44:40.134036+0000","last_peered":"2024-09-21T05:44:40.134036+0000","last_clean":"2024-09-21T05:44:40.134036+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:44:40.134036+0000","last_undegraded":"2024-09-21T05:44:40.134036+0000","last_fullsized":"2024-09-21T05:44:40.134036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139995+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:44:40.139995+0000","last_peered":"2024-09-21T05:44:40.139995+0000","last_clean":"2024-09-21T05:44:40.139995+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:44:40.139995+0000","last_undegraded":"2024-09-21T05:44:40.139995+0000","last_fullsized":"2024-09-21T05:44:40.139995+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125119+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:44:40.125119+0000","last_peered":"2024-09-21T05:44:40.125119+0000","last_clean":"2024-09-21T05:44:40.125119+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:44:40.125119+0000","last_undegraded":"2024-09-21T05:44:40.125119+0000","last_fullsized":"2024-09-21T05:44:40.125119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124242+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:44:40.124242+0000","last_peered":"2024-09-21T05:44:40.124242+0000","last_clean":"2024-09-21T05:44:40.124242+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:44:40.124242+0000","last_undegraded":"2024-09-21T05:44:40.124242+0000","last_fullsized":"2024-09-21T05:44:40.124242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137028+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:44:40.137028+0000","last_peered":"2024-09-21T05:44:40.137028+0000","last_clean":"2024-09-21T05:44:40.137028+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:44:40.137028+0000","last_undegraded":"2024-09-21T05:44:40.137028+0000","last_fullsized":"2024-09-21T05:44:40.137028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136835+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:44:40.136835+0000","last_peered":"2024-09-21T05:44:40.136835+0000","last_clean":"2024-09-21T05:44:40.136835+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:44:40.136835+0000","last_undegraded":"2024-09-21T05:44:40.136835+0000","last_fullsized":"2024-09-21T05:44:40.136835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136326+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:44:40.136326+0000","last_peered":"2024-09-21T05:44:40.136326+0000","last_clean":"2024-09-21T05:44:40.136326+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:44:40.136326+0000","last_undegraded":"2024-09-21T05:44:40.136326+0000","last_fullsized":"2024-09-21T05:44:40.136326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133696+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:44:40.133696+0000","last_peered":"2024-09-21T05:44:40.133696+0000","last_clean":"2024-09-21T05:44:40.133696+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:44:40.133696+0000","last_undegraded":"2024-09-21T05:44:40.133696+0000","last_fullsized":"2024-09-21T05:44:40.133696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135812+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:44:40.135812+0000","last_peered":"2024-09-21T05:44:40.135812+0000","last_clean":"2024-09-21T05:44:40.135812+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:44:40.135812+0000","last_undegraded":"2024-09-21T05:44:40.135812+0000","last_fullsized":"2024-09-21T05:44:40.135812+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.126209+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:44:40.126209+0000","last_peered":"2024-09-21T05:44:40.126209+0000","last_clean":"2024-09-21T05:44:40.126209+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:44:40.126209+0000","last_undegraded":"2024-09-21T05:44:40.126209+0000","last_fullsized":"2024-09-21T05:44:40.126209+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133450+0000","last_change":"2024-09-21T05:44:39.119824+0000","last_active":"2024-09-21T05:44:40.133450+0000","last_peered":"2024-09-21T05:44:40.133450+0000","last_clean":"2024-09-21T05:44:40.133450+0000","last_became_active":"2024-09-21T05:44:39.119475+0000","last_became_peered":"2024-09-21T05:44:39.119475+0000","last_unstale":"2024-09-21T05:44:40.133450+0000","last_undegraded":"2024-09-21T05:44:40.133450+0000","last_fullsized":"2024-09-21T05:44:40.133450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:29.773122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124197+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:44:40.124197+0000","last_peered":"2024-09-21T05:44:40.124197+0000","last_clean":"2024-09-21T05:44:40.124197+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:44:40.124197+0000","last_undegraded":"2024-09-21T05:44:40.124197+0000","last_fullsized":"2024-09-21T05:44:40.124197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137091+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:44:40.137091+0000","last_peered":"2024-09-21T05:44:40.137091+0000","last_clean":"2024-09-21T05:44:40.137091+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:44:40.137091+0000","last_undegraded":"2024-09-21T05:44:40.137091+0000","last_fullsized":"2024-09-21T05:44:40.137091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133517+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:44:40.133517+0000","last_peered":"2024-09-21T05:44:40.133517+0000","last_clean":"2024-09-21T05:44:40.133517+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:44:40.133517+0000","last_undegraded":"2024-09-21T05:44:40.133517+0000","last_fullsized":"2024-09-21T05:44:40.133517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138681+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:44:40.138681+0000","last_peered":"2024-09-21T05:44:40.138681+0000","last_clean":"2024-09-21T05:44:40.138681+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:44:40.138681+0000","last_undegraded":"2024-09-21T05:44:40.138681+0000","last_fullsized":"2024-09-21T05:44:40.138681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134110+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:44:40.134110+0000","last_peered":"2024-09-21T05:44:40.134110+0000","last_clean":"2024-09-21T05:44:40.134110+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:44:40.134110+0000","last_undegraded":"2024-09-21T05:44:40.134110+0000","last_fullsized":"2024-09-21T05:44:40.134110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.126101+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:44:40.126101+0000","last_peered":"2024-09-21T05:44:40.126101+0000","last_clean":"2024-09-21T05:44:40.126101+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:44:40.126101+0000","last_undegraded":"2024-09-21T05:44:40.126101+0000","last_fullsized":"2024-09-21T05:44:40.126101+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133187+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:44:40.133187+0000","last_peered":"2024-09-21T05:44:40.133187+0000","last_clean":"2024-09-21T05:44:40.133187+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:44:40.133187+0000","last_undegraded":"2024-09-21T05:44:40.133187+0000","last_fullsized":"2024-09-21T05:44:40.133187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124134+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:44:40.124134+0000","last_peered":"2024-09-21T05:44:40.124134+0000","last_clean":"2024-09-21T05:44:40.124134+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:44:40.124134+0000","last_undegraded":"2024-09-21T05:44:40.124134+0000","last_fullsized":"2024-09-21T05:44:40.124134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137199+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:44:40.137199+0000","last_peered":"2024-09-21T05:44:40.137199+0000","last_clean":"2024-09-21T05:44:40.137199+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:44:40.137199+0000","last_undegraded":"2024-09-21T05:44:40.137199+0000","last_fullsized":"2024-09-21T05:44:40.137199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125218+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:44:40.125218+0000","last_peered":"2024-09-21T05:44:40.125218+0000","last_clean":"2024-09-21T05:44:40.125218+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:44:40.125218+0000","last_undegraded":"2024-09-21T05:44:40.125218+0000","last_fullsized":"2024-09-21T05:44:40.125218+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.125790+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:44:40.125790+0000","last_peered":"2024-09-21T05:44:40.125790+0000","last_clean":"2024-09-21T05:44:40.125790+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:44:40.125790+0000","last_undegraded":"2024-09-21T05:44:40.125790+0000","last_fullsized":"2024-09-21T05:44:40.125790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.132870+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:44:40.132870+0000","last_peered":"2024-09-21T05:44:40.132870+0000","last_clean":"2024-09-21T05:44:40.132870+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:44:40.132870+0000","last_undegraded":"2024-09-21T05:44:40.132870+0000","last_fullsized":"2024-09-21T05:44:40.132870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136508+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:44:40.136508+0000","last_peered":"2024-09-21T05:44:40.136508+0000","last_clean":"2024-09-21T05:44:40.136508+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:44:40.136508+0000","last_undegraded":"2024-09-21T05:44:40.136508+0000","last_fullsized":"2024-09-21T05:44:40.136508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:01.073683+0000","last_change":"2024-09-21T05:44:37.112922+0000","last_active":"2024-09-21T05:45:01.073683+0000","last_peered":"2024-09-21T05:45:01.073683+0000","last_clean":"2024-09-21T05:45:01.073683+0000","last_became_active":"2024-09-21T05:44:37.112457+0000","last_became_peered":"2024-09-21T05:44:37.112457+0000","last_unstale":"2024-09-21T05:45:01.073683+0000","last_undegraded":"2024-09-21T05:45:01.073683+0000","last_fullsized":"2024-09-21T05:45:01.073683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:41.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139856+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:44:40.139856+0000","last_peered":"2024-09-21T05:44:40.139856+0000","last_clean":"2024-09-21T05:44:40.139856+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:44:40.139856+0000","last_undegraded":"2024-09-21T05:44:40.139856+0000","last_fullsized":"2024-09-21T05:44:40.139856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":26,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.198895+0000","last_change":"2024-09-21T05:44:37.112622+0000","last_active":"2024-09-21T05:44:50.198895+0000","last_peered":"2024-09-21T05:44:50.198895+0000","last_clean":"2024-09-21T05:44:50.198895+0000","last_became_active":"2024-09-21T05:44:37.112215+0000","last_became_peered":"2024-09-21T05:44:37.112215+0000","last_unstale":"2024-09-21T05:44:50.198895+0000","last_undegraded":"2024-09-21T05:44:50.198895+0000","last_fullsized":"2024-09-21T05:44:50.198895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:48.725835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:49.189856+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:44:49.189856+0000","last_peered":"2024-09-21T05:44:49.189856+0000","last_clean":"2024-09-21T05:44:49.189856+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:44:49.189856+0000","last_undegraded":"2024-09-21T05:44:49.189856+0000","last_fullsized":"2024-09-21T05:44:49.189856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133641+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:44:40.133641+0000","last_peered":"2024-09-21T05:44:40.133641+0000","last_clean":"2024-09-21T05:44:40.133641+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:44:40.133641+0000","last_undegraded":"2024-09-21T05:44:40.133641+0000","last_fullsized":"2024-09-21T05:44:40.133641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212702+0000","last_change":"2024-09-21T05:44:37.110808+0000","last_active":"2024-09-21T05:44:51.212702+0000","last_peered":"2024-09-21T05:44:51.212702+0000","last_clean":"2024-09-21T05:44:51.212702+0000","last_became_active":"2024-09-21T05:44:37.110376+0000","last_became_peered":"2024-09-21T05:44:37.110376+0000","last_unstale":"2024-09-21T05:44:51.212702+0000","last_undegraded":"2024-09-21T05:44:51.212702+0000","last_fullsized":"2024-09-21T05:44:51.212702+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:11.562980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135325+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:44:40.135325+0000","last_peered":"2024-09-21T05:44:40.135325+0000","last_clean":"2024-09-21T05:44:40.135325+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:44:40.135325+0000","last_undegraded":"2024-09-21T05:44:40.135325+0000","last_fullsized":"2024-09-21T05:44:40.135325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"71'4","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.074240+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:45:02.074240+0000","last_peered":"2024-09-21T05:45:02.074240+0000","last_clean":"2024-09-21T05:45:02.074240+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:45:02.074240+0000","last_undegraded":"2024-09-21T05:45:02.074240+0000","last_fullsized":"2024-09-21T05:45:02.074240+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136852+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:44:40.136852+0000","last_peered":"2024-09-21T05:44:40.136852+0000","last_clean":"2024-09-21T05:44:40.136852+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:44:40.136852+0000","last_undegraded":"2024-09-21T05:44:40.136852+0000","last_fullsized":"2024-09-21T05:44:40.136852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212070+0000","last_change":"2024-09-21T05:44:37.110361+0000","last_active":"2024-09-21T05:44:51.212070+0000","last_peered":"2024-09-21T05:44:51.212070+0000","last_clean":"2024-09-21T05:44:51.212070+0000","last_became_active":"2024-09-21T05:44:37.109912+0000","last_became_peered":"2024-09-21T05:44:37.109912+0000","last_unstale":"2024-09-21T05:44:51.212070+0000","last_undegraded":"2024-09-21T05:44:51.212070+0000","last_fullsized":"2024-09-21T05:44:51.212070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:05:00.470854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137738+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:44:40.137738+0000","last_peered":"2024-09-21T05:44:40.137738+0000","last_clean":"2024-09-21T05:44:40.137738+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:44:40.137738+0000","last_undegraded":"2024-09-21T05:44:40.137738+0000","last_fullsized":"2024-09-21T05:44:40.137738+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.197688+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:44:50.197688+0000","last_peered":"2024-09-21T05:44:50.197688+0000","last_clean":"2024-09-21T05:44:50.197688+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:44:50.197688+0000","last_undegraded":"2024-09-21T05:44:50.197688+0000","last_fullsized":"2024-09-21T05:44:50.197688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137590+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:44:40.137590+0000","last_peered":"2024-09-21T05:44:40.137590+0000","last_clean":"2024-09-21T05:44:40.137590+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:44:40.137590+0000","last_undegraded":"2024-09-21T05:44:40.137590+0000","last_fullsized":"2024-09-21T05:44:40.137590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"71'27","reported_seq":74,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.077328+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:45:02.077328+0000","last_peered":"2024-09-21T05:45:02.077328+0000","last_clean":"2024-09-21T05:45:02.077328+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:45:02.077328+0000","last_undegraded":"2024-09-21T05:45:02.077328+0000","last_fullsized":"2024-09-21T05:45:02.077328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54255,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":66,"num_write":29,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137071+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:44:40.137071+0000","last_peered":"2024-09-21T05:44:40.137071+0000","last_clean":"2024-09-21T05:44:40.137071+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:44:40.137071+0000","last_undegraded":"2024-09-21T05:44:40.137071+0000","last_fullsized":"2024-09-21T05:44:40.137071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056148+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:44:52.056148+0000","last_peered":"2024-09-21T05:44:52.056148+0000","last_clean":"2024-09-21T05:44:52.056148+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:44:52.056148+0000","last_undegraded":"2024-09-21T05:44:52.056148+0000","last_fullsized":"2024-09-21T05:44:52.056148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124014+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:44:40.124014+0000","last_peered":"2024-09-21T05:44:40.124014+0000","last_clean":"2024-09-21T05:44:40.124014+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:44:40.124014+0000","last_undegraded":"2024-09-21T05:44:40.124014+0000","last_fullsized":"2024-09-21T05:44:40.124014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.212863+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:44:51.212863+0000","last_peered":"2024-09-21T05:44:51.212863+0000","last_clean":"2024-09-21T05:44:51.212863+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:44:51.212863+0000","last_undegraded":"2024-09-21T05:44:51.212863+0000","last_fullsized":"2024-09-21T05:44:51.212863+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139564+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:44:40.139564+0000","last_peered":"2024-09-21T05:44:40.139564+0000","last_clean":"2024-09-21T05:44:40.139564+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:44:40.139564+0000","last_undegraded":"2024-09-21T05:44:40.139564+0000","last_fullsized":"2024-09-21T05:44:40.139564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.074559+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:45:02.074559+0000","last_peered":"2024-09-21T05:45:02.074559+0000","last_clean":"2024-09-21T05:45:02.074559+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:45:02.074559+0000","last_undegraded":"2024-09-21T05:45:02.074559+0000","last_fullsized":"2024-09-21T05:45:02.074559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139228+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:44:40.139228+0000","last_peered":"2024-09-21T05:44:40.139228+0000","last_clean":"2024-09-21T05:44:40.139228+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:44:40.139228+0000","last_undegraded":"2024-09-21T05:44:40.139228+0000","last_fullsized":"2024-09-21T05:44:40.139228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210953+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:44:51.210953+0000","last_peered":"2024-09-21T05:44:51.210953+0000","last_clean":"2024-09-21T05:44:51.210953+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:44:51.210953+0000","last_undegraded":"2024-09-21T05:44:51.210953+0000","last_fullsized":"2024-09-21T05:44:51.210953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137685+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:44:40.137685+0000","last_peered":"2024-09-21T05:44:40.137685+0000","last_clean":"2024-09-21T05:44:40.137685+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:44:40.137685+0000","last_undegraded":"2024-09-21T05:44:40.137685+0000","last_fullsized":"2024-09-21T05:44:40.137685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.184984+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:44:51.184984+0000","last_peered":"2024-09-21T05:44:51.184984+0000","last_clean":"2024-09-21T05:44:51.184984+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:44:51.184984+0000","last_undegraded":"2024-09-21T05:44:51.184984+0000","last_fullsized":"2024-09-21T05:44:51.184984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133386+0000","last_change":"2024-09-21T05:44:39.114163+0000","last_active":"2024-09-21T05:44:40.133386+0000","last_peered":"2024-09-21T05:44:40.133386+0000","last_clean":"2024-09-21T05:44:40.133386+0000","last_became_active":"2024-09-21T05:44:39.113799+0000","last_became_peered":"2024-09-21T05:44:39.113799+0000","last_unstale":"2024-09-21T05:44:40.133386+0000","last_undegraded":"2024-09-21T05:44:40.133386+0000","last_fullsized":"2024-09-21T05:44:40.133386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.286926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211416+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:44:51.211416+0000","last_peered":"2024-09-21T05:44:51.211416+0000","last_clean":"2024-09-21T05:44:51.211416+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:44:51.211416+0000","last_undegraded":"2024-09-21T05:44:51.211416+0000","last_fullsized":"2024-09-21T05:44:51.211416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139903+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:44:40.139903+0000","last_peered":"2024-09-21T05:44:40.139903+0000","last_clean":"2024-09-21T05:44:40.139903+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:44:40.139903+0000","last_undegraded":"2024-09-21T05:44:40.139903+0000","last_fullsized":"2024-09-21T05:44:40.139903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210590+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:44:51.210590+0000","last_peered":"2024-09-21T05:44:51.210590+0000","last_clean":"2024-09-21T05:44:51.210590+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:44:51.210590+0000","last_undegraded":"2024-09-21T05:44:51.210590+0000","last_fullsized":"2024-09-21T05:44:51.210590+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.927251+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:44:40.927251+0000","last_peered":"2024-09-21T05:44:40.927251+0000","last_clean":"2024-09-21T05:44:40.927251+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:44:40.927251+0000","last_undegraded":"2024-09-21T05:44:40.927251+0000","last_fullsized":"2024-09-21T05:44:40.927251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"71'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:00.072179+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:45:00.072179+0000","last_peered":"2024-09-21T05:45:00.072179+0000","last_clean":"2024-09-21T05:45:00.072179+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:45:00.072179+0000","last_undegraded":"2024-09-21T05:45:00.072179+0000","last_fullsized":"2024-09-21T05:45:00.072179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.140027+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:44:40.140027+0000","last_peered":"2024-09-21T05:44:40.140027+0000","last_clean":"2024-09-21T05:44:40.140027+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:44:40.140027+0000","last_undegraded":"2024-09-21T05:44:40.140027+0000","last_fullsized":"2024-09-21T05:44:40.140027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":88,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139323+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:44:40.139323+0000","last_peered":"2024-09-21T05:44:40.139323+0000","last_clean":"2024-09-21T05:44:40.139323+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:44:40.139323+0000","last_undegraded":"2024-09-21T05:44:40.139323+0000","last_fullsized":"2024-09-21T05:44:40.139323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":28,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056608+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:44:52.056608+0000","last_peered":"2024-09-21T05:44:52.056608+0000","last_clean":"2024-09-21T05:44:52.056608+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:44:52.056608+0000","last_undegraded":"2024-09-21T05:44:52.056608+0000","last_fullsized":"2024-09-21T05:44:52.056608+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.135011+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:44:40.135011+0000","last_peered":"2024-09-21T05:44:40.135011+0000","last_clean":"2024-09-21T05:44:40.135011+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:44:40.135011+0000","last_undegraded":"2024-09-21T05:44:40.135011+0000","last_fullsized":"2024-09-21T05:44:40.135011+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:49.190605+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:44:49.190605+0000","last_peered":"2024-09-21T05:44:49.190605+0000","last_clean":"2024-09-21T05:44:49.190605+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:44:49.190605+0000","last_undegraded":"2024-09-21T05:44:49.190605+0000","last_fullsized":"2024-09-21T05:44:49.190605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124412+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:44:40.124412+0000","last_peered":"2024-09-21T05:44:40.124412+0000","last_clean":"2024-09-21T05:44:40.124412+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:44:40.124412+0000","last_undegraded":"2024-09-21T05:44:40.124412+0000","last_fullsized":"2024-09-21T05:44:40.124412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:00.071323+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:45:00.071323+0000","last_peered":"2024-09-21T05:45:00.071323+0000","last_clean":"2024-09-21T05:45:00.071323+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:45:00.071323+0000","last_undegraded":"2024-09-21T05:45:00.071323+0000","last_fullsized":"2024-09-21T05:45:00.071323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.138932+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:44:40.138932+0000","last_peered":"2024-09-21T05:44:40.138932+0000","last_clean":"2024-09-21T05:44:40.138932+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:44:40.138932+0000","last_undegraded":"2024-09-21T05:44:40.138932+0000","last_fullsized":"2024-09-21T05:44:40.138932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:59.070068+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:44:59.070068+0000","last_peered":"2024-09-21T05:44:59.070068+0000","last_clean":"2024-09-21T05:44:59.070068+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:44:59.070068+0000","last_undegraded":"2024-09-21T05:44:59.070068+0000","last_fullsized":"2024-09-21T05:44:59.070068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.133684+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:44:40.133684+0000","last_peered":"2024-09-21T05:44:40.133684+0000","last_clean":"2024-09-21T05:44:40.133684+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:44:40.133684+0000","last_undegraded":"2024-09-21T05:44:40.133684+0000","last_fullsized":"2024-09-21T05:44:40.133684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136454+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:44:40.136454+0000","last_peered":"2024-09-21T05:44:40.136454+0000","last_clean":"2024-09-21T05:44:40.136454+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:44:40.136454+0000","last_undegraded":"2024-09-21T05:44:40.136454+0000","last_fullsized":"2024-09-21T05:44:40.136454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:42.530528+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:44:42.530528+0000","last_peered":"2024-09-21T05:44:42.530528+0000","last_clean":"2024-09-21T05:44:42.530528+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:44:42.530528+0000","last_undegraded":"2024-09-21T05:44:42.530528+0000","last_fullsized":"2024-09-21T05:44:42.530528+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211397+0000","last_change":"2024-09-21T05:44:37.270537+0000","last_active":"2024-09-21T05:44:51.211397+0000","last_peered":"2024-09-21T05:44:51.211397+0000","last_clean":"2024-09-21T05:44:51.211397+0000","last_became_active":"2024-09-21T05:44:37.269845+0000","last_became_peered":"2024-09-21T05:44:37.269845+0000","last_unstale":"2024-09-21T05:44:51.211397+0000","last_undegraded":"2024-09-21T05:44:51.211397+0000","last_fullsized":"2024-09-21T05:44:51.211397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:39.402650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.124501+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:44:40.124501+0000","last_peered":"2024-09-21T05:44:40.124501+0000","last_clean":"2024-09-21T05:44:40.124501+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:44:40.124501+0000","last_undegraded":"2024-09-21T05:44:40.124501+0000","last_fullsized":"2024-09-21T05:44:40.124501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"71'12","reported_seq":31,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:01.131922+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:01.131922+0000","last_peered":"2024-09-21T05:45:01.131922+0000","last_clean":"2024-09-21T05:45:01.131922+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:01.131922+0000","last_undegraded":"2024-09-21T05:45:01.131922+0000","last_fullsized":"2024-09-21T05:45:01.131922+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134274+0000","last_change":"2024-09-21T05:44:39.125685+0000","last_active":"2024-09-21T05:44:40.134274+0000","last_peered":"2024-09-21T05:44:40.134274+0000","last_clean":"2024-09-21T05:44:40.134274+0000","last_became_active":"2024-09-21T05:44:39.125308+0000","last_became_peered":"2024-09-21T05:44:39.125308+0000","last_unstale":"2024-09-21T05:44:40.134274+0000","last_undegraded":"2024-09-21T05:44:40.134274+0000","last_fullsized":"2024-09-21T05:44:40.134274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:48.599727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.211139+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:44:51.211139+0000","last_peered":"2024-09-21T05:44:51.211139+0000","last_clean":"2024-09-21T05:44:51.211139+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:44:51.211139+0000","last_undegraded":"2024-09-21T05:44:51.211139+0000","last_fullsized":"2024-09-21T05:44:51.211139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137636+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:44:40.137636+0000","last_peered":"2024-09-21T05:44:40.137636+0000","last_clean":"2024-09-21T05:44:40.137636+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:44:40.137636+0000","last_undegraded":"2024-09-21T05:44:40.137636+0000","last_fullsized":"2024-09-21T05:44:40.137636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:48.180015+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:44:48.180015+0000","last_peered":"2024-09-21T05:44:48.180015+0000","last_clean":"2024-09-21T05:44:48.180015+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:44:48.180015+0000","last_undegraded":"2024-09-21T05:44:48.180015+0000","last_fullsized":"2024-09-21T05:44:48.180015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136233+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:44:40.136233+0000","last_peered":"2024-09-21T05:44:40.136233+0000","last_clean":"2024-09-21T05:44:40.136233+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:44:40.136233+0000","last_undegraded":"2024-09-21T05:44:40.136233+0000","last_fullsized":"2024-09-21T05:44:40.136233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":26,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210212+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:44:51.210212+0000","last_peered":"2024-09-21T05:44:51.210212+0000","last_clean":"2024-09-21T05:44:51.210212+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:44:51.210212+0000","last_undegraded":"2024-09-21T05:44:51.210212+0000","last_fullsized":"2024-09-21T05:44:51.210212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.132996+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:44:40.132996+0000","last_peered":"2024-09-21T05:44:40.132996+0000","last_clean":"2024-09-21T05:44:40.132996+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:44:40.132996+0000","last_undegraded":"2024-09-21T05:44:40.132996+0000","last_fullsized":"2024-09-21T05:44:40.132996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:50.198543+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:44:50.198543+0000","last_peered":"2024-09-21T05:44:50.198543+0000","last_clean":"2024-09-21T05:44:50.198543+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:44:50.198543+0000","last_undegraded":"2024-09-21T05:44:50.198543+0000","last_fullsized":"2024-09-21T05:44:50.198543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134191+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:44:40.134191+0000","last_peered":"2024-09-21T05:44:40.134191+0000","last_clean":"2024-09-21T05:44:40.134191+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:44:40.134191+0000","last_undegraded":"2024-09-21T05:44:40.134191+0000","last_fullsized":"2024-09-21T05:44:40.134191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:48.180657+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:44:48.180657+0000","last_peered":"2024-09-21T05:44:48.180657+0000","last_clean":"2024-09-21T05:44:48.180657+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:44:48.180657+0000","last_undegraded":"2024-09-21T05:44:48.180657+0000","last_fullsized":"2024-09-21T05:44:48.180657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139404+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:44:40.139404+0000","last_peered":"2024-09-21T05:44:40.139404+0000","last_clean":"2024-09-21T05:44:40.139404+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:44:40.139404+0000","last_undegraded":"2024-09-21T05:44:40.139404+0000","last_fullsized":"2024-09-21T05:44:40.139404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.139111+0000","last_change":"2024-09-21T05:44:39.127591+0000","last_active":"2024-09-21T05:44:40.139111+0000","last_peered":"2024-09-21T05:44:40.139111+0000","last_clean":"2024-09-21T05:44:40.139111+0000","last_became_active":"2024-09-21T05:44:39.127179+0000","last_became_peered":"2024-09-21T05:44:39.127179+0000","last_unstale":"2024-09-21T05:44:40.139111+0000","last_undegraded":"2024-09-21T05:44:40.139111+0000","last_fullsized":"2024-09-21T05:44:40.139111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:55.173951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:51.210759+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:44:51.210759+0000","last_peered":"2024-09-21T05:44:51.210759+0000","last_clean":"2024-09-21T05:44:51.210759+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:44:51.210759+0000","last_undegraded":"2024-09-21T05:44:51.210759+0000","last_fullsized":"2024-09-21T05:44:51.210759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.136466+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:44:40.136466+0000","last_peered":"2024-09-21T05:44:40.136466+0000","last_clean":"2024-09-21T05:44:40.136466+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:44:40.136466+0000","last_undegraded":"2024-09-21T05:44:40.136466+0000","last_fullsized":"2024-09-21T05:44:40.136466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:52.056120+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:44:52.056120+0000","last_peered":"2024-09-21T05:44:52.056120+0000","last_clean":"2024-09-21T05:44:52.056120+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:44:52.056120+0000","last_undegraded":"2024-09-21T05:44:52.056120+0000","last_fullsized":"2024-09-21T05:44:52.056120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.137929+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:44:40.137929+0000","last_peered":"2024-09-21T05:44:40.137929+0000","last_clean":"2024-09-21T05:44:40.137929+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:44:40.137929+0000","last_undegraded":"2024-09-21T05:44:40.137929+0000","last_fullsized":"2024-09-21T05:44:40.137929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.140161+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:44:40.140161+0000","last_peered":"2024-09-21T05:44:40.140161+0000","last_clean":"2024-09-21T05:44:40.140161+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:44:40.140161+0000","last_undegraded":"2024-09-21T05:44:40.140161+0000","last_fullsized":"2024-09-21T05:44:40.140161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:44:40.134024+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:44:40.134024+0000","last_peered":"2024-09-21T05:44:40.134024+0000","last_clean":"2024-09-21T05:44:40.134024+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:44:40.134024+0000","last_undegraded":"2024-09-21T05:44:40.134024+0000","last_fullsized":"2024-09-21T05:44:40.134024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-21T05:45:02.073490+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:45:02.073490+0000","last_peered":"2024-09-21T05:45:02.073490+0000","last_clean":"2024-09-21T05:45:02.073490+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:02.073490+0000","last_undegraded":"2024-09-21T05:45:02.073490+0000","last_fullsized":"2024-09-21T05:45:02.073490+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61068,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":145,"num_read_kb":141,"num_write":234,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":183204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874258,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36100,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709052,"statfs":{"total":95995035648,"available":95958069248,"internally_reserved":0,"allocated":774144,"data_stored":381583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037782,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712532,"statfs":{"total":95995035648,"available":95961632768,"internally_reserved":0,"allocated":1470464,"data_stored":1026476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201307,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36124,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709028,"statfs":{"total":95995035648,"available":95958044672,"internally_reserved":0,"allocated":798720,"data_stored":381122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":794624,"data_stored":381500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561060,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708448,"statfs":{"total":95995035648,"available":95957450752,"internally_reserved":0,"allocated":1392640,"data_stored":973627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724585,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":782336,"data_stored":380839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888110,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1404928,"data_stored":973305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":782336,"data_stored":380700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":798720,"data_stored":380664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":847872,"data_stored":427566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542210,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":827392,"data_stored":426955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":802816,"data_stored":382569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":55991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:45:07.183 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T05:45:07.184 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T05:45:07.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T05:45:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:07.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:07 smithi181 ceph-mon[34819]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:07 smithi028 ceph-mon[31227]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:07 smithi129 ceph-mon[34885]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 -- 172.21.15.28:0/1872561145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1062d0 msgr2=0x7f0c2c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 --2- 172.21.15.28:0/1872561145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1066e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009900 tx=0x7f0c2002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 -- 172.21.15.28:0/1872561145 shutdown_connections 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 --2- 172.21.15.28:0/1872561145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c106c20 0x7f0c2c1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 --2- 172.21.15.28:0/1872561145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1066e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 --2- 172.21.15.28:0/1872561145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 -- 172.21.15.28:0/1872561145 >> 172.21.15.28:0/1872561145 conn(0x7f0c2c079f40 msgr2=0x7f0c2c07a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.877+0000 7f0c319b9640 1 -- 172.21.15.28:0/1872561145 shutdown_connections 2024-09-21T05:45:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.878+0000 7f0c319b9640 1 -- 172.21.15.28:0/1872561145 wait complete. 2024-09-21T05:45:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.878+0000 7f0c319b9640 1 Processor -- start 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.878+0000 7f0c319b9640 1 -- start start 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c106c20 0x7f0c2c103b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c113af0 con 0x7f0c2c1020e0 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c113970 con 0x7f0c2c106c20 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c319b9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c113c70 con 0x7f0c2c1062d0 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54520/0 (socket says 172.21.15.28:54520) 2024-09-21T05:45:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 -- 172.21.15.28:0/2165496909 learned_addr learned my addr 172.21.15.28:0/2165496909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2a7fc640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2b7fe640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c106c20 0x7f0c2c103b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 -- 172.21.15.28:0/2165496909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c1062d0 msgr2=0x7f0c2c1035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 -- 172.21.15.28:0/2165496909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c106c20 msgr2=0x7f0c2c103b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c106c20 0x7f0c2c103b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 -- 172.21.15.28:0/2165496909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c1a8820 con 0x7f0c2c1020e0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.879+0000 7f0c2affd640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00d770 tx=0x7f0c1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.880+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c017070 con 0x7f0c2c1020e0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.880+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c1c009c30 con 0x7f0c2c1020e0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.880+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c003c90 con 0x7f0c2c1020e0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.880+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2c1a8b10 con 0x7f0c2c1020e0 2024-09-21T05:45:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.880+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2c1a9070 con 0x7f0c2c1020e0 2024-09-21T05:45:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.881+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c2c1024f0 con 0x7f0c2c1020e0 2024-09-21T05:45:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.882+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f0c1c003e30 con 0x7f0c2c1020e0 2024-09-21T05:45:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.883+0000 7f0c309b7640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 0x7f0c140b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.884+0000 7f0c2a7fc640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 0x7f0c140b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.884+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f0c1c0e7140 con 0x7f0c2c1020e0 2024-09-21T05:45:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.884+0000 7f0c2a7fc640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 0x7f0c140b8b80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c20005940 tx=0x7f0c200058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:07.887+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0c1c009de0 con 0x7f0c2c1020e0 2024-09-21T05:45:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.148+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f0c2c02e640 con 0x7f0c2c1020e0 2024-09-21T05:45:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.151+0000 7f0c309b7640 1 -- 172.21.15.28:0/2165496909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f0c1c0b1cf0 con 0x7f0c2c1020e0 2024-09-21T05:45:08.153 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-21T05:45:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBkXe5mZCnvCBAAlhXh74JOBhEzdTT9ifHytg== 2024-09-21T05:45:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 msgr2=0x7f0c140b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 0x7f0c140b8b80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c20005940 tx=0x7f0c200058d0 comp rx=0 tx=0).stop 2024-09-21T05:45:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 msgr2=0x7f0c2c103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00d770 tx=0x7f0c1c00dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 shutdown_connections 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c2c106c20 0x7f0c2c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c2c1062d0 0x7f0c2c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0c140b6670 0x7f0c140b8b80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 --2- 172.21.15.28:0/2165496909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c1020e0 0x7f0c2c103080 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.153+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 >> 172.21.15.28:0/2165496909 conn(0x7f0c2c079f40 msgr2=0x7f0c2c107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.154+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 shutdown_connections 2024-09-21T05:45:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:08.154+0000 7f0c319b9640 1 -- 172.21.15.28:0/2165496909 wait complete. 2024-09-21T05:45:08.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:08 smithi028 ceph-mon[31227]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-21T05:45:08.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:08 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T05:45:08.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:08 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T05:45:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:08 smithi181 ceph-mon[34819]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-21T05:45:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:08 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T05:45:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:08 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T05:45:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:08 smithi129 ceph-mon[34885]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-21T05:45:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:08 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T05:45:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:08 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2165496909' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T05:45:08.931 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:45:08.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T05:45:08.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T05:45:08.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T05:45:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 -- 172.21.15.28:0/3659881988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c102ae0 msgr2=0x7fe13c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 --2- 172.21.15.28:0/3659881988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c102ae0 0x7fe13c102ef0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe138009900 tx=0x7fe13802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 -- 172.21.15.28:0/3659881988 shutdown_connections 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 --2- 172.21.15.28:0/3659881988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c108d20 0x7fe13c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 --2- 172.21.15.28:0/3659881988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c1083d0 0x7fe13c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 --2- 172.21.15.28:0/3659881988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c102ae0 0x7fe13c102ef0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 -- 172.21.15.28:0/3659881988 >> 172.21.15.28:0/3659881988 conn(0x7fe13c079760 msgr2=0x7fe13c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 -- 172.21.15.28:0/3659881988 shutdown_connections 2024-09-21T05:45:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.826+0000 7fe14412c640 1 -- 172.21.15.28:0/3659881988 wait complete. 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 Processor -- start 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 -- start start 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 0x7fe13c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c108d20 0x7fe13c19bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1129f0 con 0x7fe13c1083d0 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c112870 con 0x7fe13c102ae0 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe14412c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c112b70 con 0x7fe13c108d20 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe1416c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.827+0000 7fe1416c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54538/0 (socket says 172.21.15.28:54538) 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 -- 172.21.15.28:0/2731485823 learned_addr learned my addr 172.21.15.28:0/2731485823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe141ec8640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 0x7fe13c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1426c9640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c108d20 0x7fe13c19bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 -- 172.21.15.28:0/2731485823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c108d20 msgr2=0x7fe13c19bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c108d20 0x7fe13c19bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 -- 172.21.15.28:0/2731485823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 msgr2=0x7fe13c19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 0x7fe13c19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 -- 172.21.15.28:0/2731485823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c1a6670 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe141ec8640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 0x7fe13c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe1416c7640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00b450 tx=0x7fe12c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c015020 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c1a6960 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe12c011d40 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c00f340 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.828+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c1a6ea0 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.829+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13c103d10 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.831+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fe12c018020 con 0x7fe13c1083d0 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.831+0000 7fe132ffd640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 0x7fe1280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.832+0000 7fe141ec8640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 0x7fe1280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.832+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7fe12c0e7530 con 0x7fe13c1083d0 2024-09-21T05:45:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.832+0000 7fe141ec8640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 0x7fe1280b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe138000c00 tx=0x7fe138031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:09.834+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe12c01f020 con 0x7fe13c1083d0 2024-09-21T05:45:10.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.025+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fe13c06bfb0 con 0x7fe13c1083d0 2024-09-21T05:45:10.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.026+0000 7fe132ffd640 1 -- 172.21.15.28:0/2731485823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v58) ==== 86+0+133 (secure 0 0 0) 0x7fe13c06bfb0 con 0x7fe13c1083d0 2024-09-21T05:45:10.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 msgr2=0x7fe1280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 0x7fe1280b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe138000c00 tx=0x7fe138031400 comp rx=0 tx=0).stop 2024-09-21T05:45:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 msgr2=0x7fe13c19b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00b450 tx=0x7fe12c00b910 comp rx=0 tx=0).stop 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 shutdown_connections 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe13c108d20 0x7fe13c19bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1280b65f0 0x7fe1280b8b00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe13c1083d0 0x7fe13c19b8a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 --2- 172.21.15.28:0/2731485823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe13c102ae0 0x7fe13c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 >> 172.21.15.28:0/2731485823 conn(0x7fe13c079760 msgr2=0x7fe13c101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 shutdown_connections 2024-09-21T05:45:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:10.028+0000 7fe14412c640 1 -- 172.21.15.28:0/2731485823 wait complete. 2024-09-21T05:45:10.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:10 smithi028 ceph-mon[31227]: pgmap v267: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-09-21T05:45:10.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:10 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2731485823' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T05:45:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:10 smithi181 ceph-mon[34819]: pgmap v267: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-09-21T05:45:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:10 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2731485823' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T05:45:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:10 smithi129 ceph-mon[34885]: pgmap v267: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-09-21T05:45:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:10 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2731485823' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T05:45:10.705 INFO:teuthology.orchestra.run.smithi028.stdout:[{"version":58,"timestamp":"2024-09-21T05:44:29.366857+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-09-21T05:45:10.705 INFO:tasks.ceph_manager:config epoch is 58 2024-09-21T05:45:10.705 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T05:45:10.706 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T05:45:10.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T05:45:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.432+0000 7f05887da640 1 -- 172.21.15.28:0/2348884614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0580103b60 msgr2=0x7f0580103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.432+0000 7f05887da640 1 --2- 172.21.15.28:0/2348884614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0580103b60 0x7f0580103f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f057c009900 tx=0x7f057c02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 -- 172.21.15.28:0/2348884614 shutdown_connections 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 --2- 172.21.15.28:0/2348884614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580109df0 0x7f0580110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 --2- 172.21.15.28:0/2348884614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05801094a0 0x7f05801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 --2- 172.21.15.28:0/2348884614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0580103b60 0x7f0580103f70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 -- 172.21.15.28:0/2348884614 >> 172.21.15.28:0/2348884614 conn(0x7f058007b850 msgr2=0x7f058007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 -- 172.21.15.28:0/2348884614 shutdown_connections 2024-09-21T05:45:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 -- 172.21.15.28:0/2348884614 wait complete. 2024-09-21T05:45:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 Processor -- start 2024-09-21T05:45:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.433+0000 7f05887da640 1 -- start start 2024-09-21T05:45:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580103b60 0x7f058019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0580109df0 0x7f058019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0580113ab0 con 0x7f05801094a0 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0580113930 con 0x7f0580109df0 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05887da640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0580113c30 con 0x7f0580103b60 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54552/0 (socket says 172.21.15.28:54552) 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 -- 172.21.15.28:0/4166740733 learned_addr learned my addr 172.21.15.28:0/4166740733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0586576640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580103b60 0x7f058019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0586d77640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0580109df0 0x7f058019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 -- 172.21.15.28:0/4166740733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580103b60 msgr2=0x7f058019a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580103b60 0x7f058019a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 -- 172.21.15.28:0/4166740733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0580109df0 msgr2=0x7f058019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0580109df0 0x7f058019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 -- 172.21.15.28:0/4166740733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05801a03b0 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f0585d75640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f057000d740 tx=0x7f057000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.434+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570017070 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.435+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f057000de50 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.435+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05801a06a0 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.435+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057000f620 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.435+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05801a6ec0 con 0x7f05801094a0 2024-09-21T05:45:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.436+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0580103f70 con 0x7f05801094a0 2024-09-21T05:45:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.438+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f057001a070 con 0x7f05801094a0 2024-09-21T05:45:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.439+0000 7f05777fe640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 0x7f056c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.439+0000 7f0586576640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 0x7f056c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.439+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f05700e6df0 con 0x7f05801094a0 2024-09-21T05:45:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.439+0000 7f0586576640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 0x7f056c0b8b30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f057c031440 tx=0x7f057c009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.442+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f05700b1920 con 0x7f05801094a0 2024-09-21T05:45:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.681+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f058010bbc0 con 0x7f05801094a0 2024-09-21T05:45:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.686+0000 7f05777fe640 1 -- 172.21.15.28:0/4166740733 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v22) ==== 74+0+296453 (secure 0 0 0) 0x7f05700b6860 con 0x7f05801094a0 2024-09-21T05:45:11.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 msgr2=0x7f056c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 0x7f056c0b8b30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f057c031440 tx=0x7f057c009560 comp rx=0 tx=0).stop 2024-09-21T05:45:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 msgr2=0x7f058019aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f057000d740 tx=0x7f057000dc00 comp rx=0 tx=0).stop 2024-09-21T05:45:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 shutdown_connections 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0580109df0 0x7f058019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f056c0b6620 0x7f056c0b8b30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05801094a0 0x7f058019aaa0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 --2- 172.21.15.28:0/4166740733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0580103b60 0x7f058019a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.689+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 >> 172.21.15.28:0/4166740733 conn(0x7f058007b850 msgr2=0x7f05801075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.690+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 shutdown_connections 2024-09-21T05:45:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:11.690+0000 7f05887da640 1 -- 172.21.15.28:0/4166740733 wait complete. 2024-09-21T05:45:12.319 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"flags":0,"active_gid":14223,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4211147350},{"type":"v1","addr":"172.21.15.28:6801","nonce":4211147350}]},"active_addr":"172.21.15.28:6801/4211147350","active_change":"2024-09-21T05:38:12.622330+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24148,"name":"z","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24164,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3789081143}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2980400559}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":728118779}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3401390844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2885885992}]}]} 2024-09-21T05:45:12.325 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T05:45:12.325 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T05:45:12.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:45:12.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:12 smithi181 ceph-mon[34819]: pgmap v268: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.5 KiB/s wr, 11 op/s 2024-09-21T05:45:12.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:12 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4166740733' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T05:45:12.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:12 smithi028 ceph-mon[31227]: pgmap v268: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.5 KiB/s wr, 11 op/s 2024-09-21T05:45:12.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:12 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4166740733' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T05:45:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:12 smithi129 ceph-mon[34885]: pgmap v268: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.5 KiB/s wr, 11 op/s 2024-09-21T05:45:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:12 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4166740733' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 -- 172.21.15.28:0/3682884717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac8102a60 msgr2=0x7f5ac8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/3682884717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac8102a60 0x7f5ac8102e70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009ca0 tx=0x7f5ab802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 -- 172.21.15.28:0/3682884717 shutdown_connections 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/3682884717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 0x7f5ac810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/3682884717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac81083a0 0x7f5ac81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/3682884717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac8102a60 0x7f5ac8102e70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 -- 172.21.15.28:0/3682884717 >> 172.21.15.28:0/3682884717 conn(0x7f5ac8079730 msgr2=0x7f5ac8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 -- 172.21.15.28:0/3682884717 shutdown_connections 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.075+0000 7f5acdc3d640 1 -- 172.21.15.28:0/3682884717 wait complete. 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 Processor -- start 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 -- start start 2024-09-21T05:45:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 0x7f5ac810ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 0x7f5ac8110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8113a80 con 0x7f5ac81083a0 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8113900 con 0x7f5ac8102a60 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.076+0000 7f5acdc3d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8113c00 con 0x7f5ac8108cf0 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54576/0 (socket says 172.21.15.28:54576) 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 -- 172.21.15.28:0/1850849449 learned_addr learned my addr 172.21.15.28:0/1850849449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac77fe640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 0x7f5ac810ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac7fff640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 0x7f5ac8110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 -- 172.21.15.28:0/1850849449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 msgr2=0x7f5ac8110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 0x7f5ac8110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 -- 172.21.15.28:0/1850849449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 msgr2=0x7f5ac810ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 0x7f5ac810ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 -- 172.21.15.28:0/1850849449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac81a8710 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac77fe640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 0x7f5ac810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac6ffd640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00d770 tx=0x7f5abc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc017070 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac81a8a00 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5abc009c30 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.077+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc00f620 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.078+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac81a8f90 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.079+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8102e70 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.080+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f5abc01a070 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.081+0000 7f5ac4ff9640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 0x7f5ab00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.081+0000 7f5ac77fe640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 0x7f5ab00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.081+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8194+0+0 (secure 0 0 0) 0x7f5abc0e7060 con 0x7f5ac81083a0 2024-09-21T05:45:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.081+0000 7f5ac77fe640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 0x7f5ab00b8b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80094e0 tx=0x7f5ab80093f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.085+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5abc009de0 con 0x7f5ac81083a0 2024-09-21T05:45:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.285+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ac802e640 con 0x7f5ac81083a0 2024-09-21T05:45:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.286+0000 7f5ac4ff9640 1 -- 172.21.15.28:0/1850849449 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+23122 (secure 0 0 0) 0x7f5abc092e20 con 0x7f5ac81083a0 2024-09-21T05:45:13.289 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:13.289 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":72,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:13.242671+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 msgr2=0x7f5ab00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 0x7f5ab00b8b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80094e0 tx=0x7f5ab80093f0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 msgr2=0x7f5ac81104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00d770 tx=0x7f5abc00dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 shutdown_connections 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ac8108cf0 0x7f5ac8110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5ab00b65f0 0x7f5ab00b8b00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ac81083a0 0x7f5ac81104f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 --2- 172.21.15.28:0/1850849449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ac8102a60 0x7f5ac810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 >> 172.21.15.28:0/1850849449 conn(0x7f5ac8079730 msgr2=0x7f5ac81013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 shutdown_connections 2024-09-21T05:45:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:13.289+0000 7f5acdc3d640 1 -- 172.21.15.28:0/1850849449 wait complete. 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]} : dispatch 2024-09-21T05:45:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:13 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T05:45:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]} : dispatch 2024-09-21T05:45:13.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:13 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]} : dispatch 2024-09-21T05:45:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T05:45:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]} : dispatch 2024-09-21T05:45:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]} : dispatch 2024-09-21T05:45:13.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:13 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]} : dispatch 2024-09-21T05:45:13.862 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T05:45:13.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:45:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:14.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: pgmap v269: 97 pgs: 97 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-21T05:45:14.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T05:45:14.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-21T05:45:14.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2024-09-21T05:45:14.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T05:45:14.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:14 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1850849449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: pgmap v269: 97 pgs: 97 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-21T05:45:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T05:45:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:14 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1850849449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 -- 172.21.15.28:0/1900127566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0103b80 msgr2=0x7f45a0103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 --2- 172.21.15.28:0/1900127566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0103b80 0x7f45a0103f90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4590009900 tx=0x7f459002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 -- 172.21.15.28:0/1900127566 shutdown_connections 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 --2- 172.21.15.28:0/1900127566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a0109e10 0x7f45a0110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 --2- 172.21.15.28:0/1900127566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a01094c0 0x7f45a01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 --2- 172.21.15.28:0/1900127566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0103b80 0x7f45a0103f90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 -- 172.21.15.28:0/1900127566 >> 172.21.15.28:0/1900127566 conn(0x7f45a00ff610 msgr2=0x7f45a0101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 -- 172.21.15.28:0/1900127566 shutdown_connections 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.575+0000 7f45a6024640 1 -- 172.21.15.28:0/1900127566 wait complete. 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.576+0000 7f45a6024640 1 Processor -- start 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.576+0000 7f45a6024640 1 -- start start 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.576+0000 7f45a6024640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 0x7f45a019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.576+0000 7f45a6024640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a01094c0 0x7f45a019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.576+0000 7f45a6024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f45a6024640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113ad0 con 0x7f45a0109e10 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f45a6024640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113950 con 0x7f45a0103b80 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f45a6024640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0113c50 con 0x7f45a01094c0 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 0x7f45a019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54592/0 (socket says 172.21.15.28:54592) 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 0x7f45a019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:50890/0 (socket says 172.21.15.28:50890) 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 -- 172.21.15.28:0/728337510 learned_addr learned my addr 172.21.15.28:0/728337510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f4596ffd640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a01094c0 0x7f45a019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 -- 172.21.15.28:0/728337510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a01094c0 msgr2=0x7f45a019aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a01094c0 0x7f45a019aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 -- 172.21.15.28:0/728337510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 msgr2=0x7f45a019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 0x7f45a019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 -- 172.21.15.28:0/728337510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a01a0490 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459ffff640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f459800eaa0 tx=0x7f459800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459800cdb0 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4598005020 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01a0780 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.577+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45980116d0 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.578+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a6ee0 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.579+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a0104e00 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.580+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f45980051d0 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.581+0000 7f459d7fa640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 0x7f458c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.581+0000 7f459f7fe640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 0x7f458c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.582+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f45980e7ad0 con 0x7f45a0109e10 2024-09-21T05:45:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.582+0000 7f459f7fe640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 0x7f458c0b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4590031440 tx=0x7f4590009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.585+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4598016030 con 0x7f45a0109e10 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: pgmap v269: 97 pgs: 97 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-09-21T05:45:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-09-21T05:45:14.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T05:45:14.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-21T05:45:14.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-21T05:45:14.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T05:45:14.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:14 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1850849449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.775+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45a019b650 con 0x7f45a0109e10 2024-09-21T05:45:14.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.776+0000 7f459d7fa640 1 -- 172.21.15.28:0/728337510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+23122 (secure 0 0 0) 0x7f45980b2480 con 0x7f45a0109e10 2024-09-21T05:45:14.779 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:14.779 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:14.248651+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 msgr2=0x7f458c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 0x7f458c0b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4590031440 tx=0x7f4590009560 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 msgr2=0x7f45a019fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f459800eaa0 tx=0x7f459800ef60 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 shutdown_connections 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f458c0b65f0 0x7f458c0b8b00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a0109e10 0x7f45a019fd60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45a01094c0 0x7f45a019aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 --2- 172.21.15.28:0/728337510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45a0103b80 0x7f45a019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 >> 172.21.15.28:0/728337510 conn(0x7f45a00ff610 msgr2=0x7f45a0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 shutdown_connections 2024-09-21T05:45:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:14.779+0000 7f45a6024640 1 -- 172.21.15.28:0/728337510 wait complete. 2024-09-21T05:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:15 smithi028 ceph-mon[31227]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T05:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:15 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/728337510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:15.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T05:45:15.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T05:45:15.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T05:45:15.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T05:45:15.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T05:45:15.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T05:45:15.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T05:45:15.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T05:45:15.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.8 flush_pg_stats 2024-09-21T05:45:15.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.9 flush_pg_stats 2024-09-21T05:45:15.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.10 flush_pg_stats 2024-09-21T05:45:15.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell osd.11 flush_pg_stats 2024-09-21T05:45:15.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:15 smithi181 ceph-mon[34819]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T05:45:15.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:15 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/728337510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:15.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:15 smithi129 ceph-mon[34885]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T05:45:15.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:15 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/728337510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.077 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:16 smithi181 ceph-mon[34819]: pgmap v272: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T05:45:16.554 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:16 smithi028 ceph-mon[31227]: pgmap v272: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T05:45:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:16 smithi129 ceph-mon[34885]: pgmap v272: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 -- 172.21.15.28:0/1784808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc01094b0 msgr2=0x7fdfc01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/1784808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc01094b0 0x7fdfc01098c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009900 tx=0x7fdfb402a410 comp rx=0 tx=0).stop 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 -- 172.21.15.28:0/1784808904 shutdown_connections 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/1784808904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0109e00 0x7fdfc0110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/1784808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc01094b0 0x7fdfc01098c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/1784808904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc0101f70 0x7fdfc0102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 -- 172.21.15.28:0/1784808904 >> 172.21.15.28:0/1784808904 conn(0x7fdfc0077c60 msgr2=0x7fdfc00780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 -- 172.21.15.28:0/1784808904 shutdown_connections 2024-09-21T05:45:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.769+0000 7fdfc5be7640 1 -- 172.21.15.28:0/1784808904 wait complete. 2024-09-21T05:45:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.770+0000 7fdfc5be7640 1 Processor -- start 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.770+0000 7fdfc5be7640 1 -- start start 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0101f70 0x7fdfc019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc01094b0 0x7fdfc019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 0x7fdfc019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc0113ac0 con 0x7fdfc0109e00 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc0113940 con 0x7fdfc0101f70 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfc5be7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc0113c40 con 0x7fdfc01094b0 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 0x7fdfc019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 0x7fdfc019fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43886/0 (socket says 172.21.15.28:43886) 2024-09-21T05:45:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbffff640 1 -- 172.21.15.28:0/3022434864 learned_addr learned my addr 172.21.15.28:0/3022434864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0101f70 0x7fdfc019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbeffd640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc01094b0 0x7fdfc019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 -- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc01094b0 msgr2=0x7fdfc019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc01094b0 0x7fdfc019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 -- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 msgr2=0x7fdfc019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 0x7fdfc019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbf7fe640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc01a0340 con 0x7fdfc0101f70 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.771+0000 7fdfbffff640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc0109e00 0x7fdfc019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfbf7fe640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0101f70 0x7fdfc019a490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000d6d0 tx=0x7fdfb000db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0017070 con 0x7fdfc0101f70 2024-09-21T05:45:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc01a0630 con 0x7fdfc0101f70 2024-09-21T05:45:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb0009820 con 0x7fdfc0101f70 2024-09-21T05:45:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc01a6e50 con 0x7fdfc0101f70 2024-09-21T05:45:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.772+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0004c40 con 0x7fdfc0101f70 2024-09-21T05:45:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.773+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdfc0063b50 con 0x7fdfc0101f70 2024-09-21T05:45:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.775+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fdfb0005220 con 0x7fdfc0101f70 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbcff9640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfa80b66a0 0x7fdfa80b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbeffd640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfa80b66a0 0x7fdfa80b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdfb00e7510 con 0x7fdfc0101f70 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbcff9640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] conn(0x7fdfa80c0510 0x7fdfa80c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfb0021e50 con 0x7fdfa80c0510 2024-09-21T05:45:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.776+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fdfb00e7960 con 0x7fdfc0101f70 2024-09-21T05:45:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.777+0000 7fdfbeffd640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfa80b66a0 0x7fdfa80b8bb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009230 tx=0x7fdfb4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.777+0000 7fdfbffff640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] conn(0x7fdfa80c0510 0x7fdfa80c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.777+0000 7fdfbffff640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] conn(0x7fdfa80c0510 0x7fdfa80c2a00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:16.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.778+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== osd.5 v2:172.21.15.181:6808/1579505960 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfb0021e50 con 0x7fdfa80c0510 2024-09-21T05:45:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.796+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 --> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdfc0063f80 con 0x7fdfa80c0510 2024-09-21T05:45:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.797+0000 7fdfbcff9640 1 -- 172.21.15.28:0/3022434864 <== osd.5 v2:172.21.15.181:6808/1579505960 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdfc0063f80 con 0x7fdfa80c0510 2024-09-21T05:45:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.797+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] conn(0x7fdfa80c0510 msgr2=0x7fdfa80c2a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.797+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] conn(0x7fdfa80c0510 0x7fdfa80c2a00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfa80b66a0 msgr2=0x7fdfa80b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfa80b66a0 0x7fdfa80b8bb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009230 tx=0x7fdfb4031400 comp rx=0 tx=0).stop 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0101f70 msgr2=0x7fdfc019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 --2- 172.21.15.28:0/3022434864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc0101f70 0x7fdfc019a490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000d6d0 tx=0x7fdfb000db90 comp rx=0 tx=0).stop 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfbffff640 1 -- 172.21.15.28:0/3022434864 reap_dead start 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 shutdown_connections 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 >> 172.21.15.28:0/3022434864 conn(0x7fdfc0077c60 msgr2=0x7fdfc01054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 shutdown_connections 2024-09-21T05:45:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:16.798+0000 7fdfc5be7640 1 -- 172.21.15.28:0/3022434864 wait complete. 2024-09-21T05:45:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:17.551 INFO:teuthology.orchestra.run.smithi028.stdout:146028888113 2024-09-21T05:45:17.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 -- 172.21.15.28:0/1987962825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc1083a0 msgr2=0x7fbcdc1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 --2- 172.21.15.28:0/1987962825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc1083a0 0x7fbcdc1087b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009900 tx=0x7fbccc02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 -- 172.21.15.28:0/1987962825 shutdown_connections 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 --2- 172.21.15.28:0/1987962825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 --2- 172.21.15.28:0/1987962825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc1083a0 0x7fbcdc1087b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 --2- 172.21.15.28:0/1987962825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcdc102a60 0x7fbcdc102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 -- 172.21.15.28:0/1987962825 >> 172.21.15.28:0/1987962825 conn(0x7fbcdc079730 msgr2=0x7fbcdc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 -- 172.21.15.28:0/1987962825 shutdown_connections 2024-09-21T05:45:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.113+0000 7fbce429d640 1 -- 172.21.15.28:0/1987962825 wait complete. 2024-09-21T05:45:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 Processor -- start 2024-09-21T05:45:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 -- start start 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcdc102a60 0x7fbcdc19b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcdc1083a0 0x7fbcdc19b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc112960 con 0x7fbcdc108cf0 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc1127e0 con 0x7fbcdc1083a0 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce429d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc112ae0 con 0x7fbcdc102a60 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc19bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43912/0 (socket says 172.21.15.28:43912) 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 -- 172.21.15.28:0/3487451065 learned_addr learned my addr 172.21.15.28:0/3487451065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce2039640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcdc102a60 0x7fbcdc19b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce1838640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcdc1083a0 0x7fbcdc19b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 -- 172.21.15.28:0/3487451065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcdc102a60 msgr2=0x7fbcdc19b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcdc102a60 0x7fbcdc19b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 -- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcdc1083a0 msgr2=0x7fbcdc19b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.114+0000 7fbce283a640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcdc1083a0 0x7fbcdc19b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbce283a640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc1a64e0 con 0x7fbcdc108cf0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbce283a640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc19bc50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400eaa0 tx=0x7fbcd400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400cdb0 con 0x7fbcdc108cf0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd4005020 con 0x7fbcdc108cf0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40116d0 con 0x7fbcdc108cf0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcdc1a67d0 con 0x7fbcdc108cf0 2024-09-21T05:45:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.115+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcdc1a6c90 con 0x7fbcdc108cf0 2024-09-21T05:45:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.116+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbcdc063b50 con 0x7fbcdc108cf0 2024-09-21T05:45:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.118+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fbcd40051d0 con 0x7fbcdc108cf0 2024-09-21T05:45:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.118+0000 7fbcd37fe640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcc80b6670 0x7fbcc80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbce2039640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcc80b6670 0x7fbcc80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbcd40e7b10 con 0x7fbcdc108cf0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbcd37fe640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] conn(0x7fbcc80c04e0 0x7fbcc80c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcc80c30a0 con 0x7fbcc80c04e0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fbcd40e7f60 con 0x7fbcdc108cf0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbce2039640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcc80b6670 0x7fbcc80b8b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc06ba20 tx=0x7fbcd800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbce1838640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] conn(0x7fbcc80c04e0 0x7fbcc80c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.119+0000 7fbce1838640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] conn(0x7fbcc80c04e0 0x7fbcc80c29d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.120+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== osd.4 v2:172.21.15.129:6808/645644998 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcc80c30a0 con 0x7fbcc80c04e0 2024-09-21T05:45:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.134+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 --> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbcdc064020 con 0x7fbcc80c04e0 2024-09-21T05:45:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.134+0000 7fbcd37fe640 1 -- 172.21.15.28:0/3487451065 <== osd.4 v2:172.21.15.129:6808/645644998 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbcdc064020 con 0x7fbcc80c04e0 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] conn(0x7fbcc80c04e0 msgr2=0x7fbcc80c29d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] conn(0x7fbcc80c04e0 0x7fbcc80c29d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcc80b6670 msgr2=0x7fbcc80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbcc80b6670 0x7fbcc80b8b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc06ba20 tx=0x7fbcd800b390 comp rx=0 tx=0).stop 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 msgr2=0x7fbcdc19bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 --2- 172.21.15.28:0/3487451065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcdc108cf0 0x7fbcdc19bc50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400eaa0 tx=0x7fbcd400ef60 comp rx=0 tx=0).stop 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce283a640 1 -- 172.21.15.28:0/3487451065 reap_dead start 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 shutdown_connections 2024-09-21T05:45:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 >> 172.21.15.28:0/3487451065 conn(0x7fbcdc079730 msgr2=0x7fbcdc104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 shutdown_connections 2024-09-21T05:45:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:18.135+0000 7fbce429d640 1 -- 172.21.15.28:0/3487451065 wait complete. 2024-09-21T05:45:18.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:18 smithi181 ceph-mon[34819]: pgmap v273: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T05:45:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:18 smithi028 ceph-mon[31227]: pgmap v273: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T05:45:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:18 smithi129 ceph-mon[34885]: pgmap v273: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 -- 172.21.15.28:0/3517558927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 msgr2=0x7f65741098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 --2- 172.21.15.28:0/3517558927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f65741098d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6568009900 tx=0x7f656802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 -- 172.21.15.28:0/3517558927 shutdown_connections 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 --2- 172.21.15.28:0/3517558927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6574109e10 0x7f6574110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 --2- 172.21.15.28:0/3517558927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f65741098d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 --2- 172.21.15.28:0/3517558927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6574103b80 0x7f6574103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 -- 172.21.15.28:0/3517558927 >> 172.21.15.28:0/3517558927 conn(0x7f65740ff610 msgr2=0x7f6574101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 -- 172.21.15.28:0/3517558927 shutdown_connections 2024-09-21T05:45:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.105+0000 7f657c723640 1 -- 172.21.15.28:0/3517558927 wait complete. 2024-09-21T05:45:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 Processor -- start 2024-09-21T05:45:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 -- start start 2024-09-21T05:45:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6574103b80 0x7f657419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f657419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6574109e10 0x7f657419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6574113ad0 con 0x7f65741094c0 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6574113950 con 0x7f6574109e10 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657c723640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6574113c50 con 0x7f6574103b80 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f6579cbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f657419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f6579cbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f657419aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43932/0 (socket says 172.21.15.28:43932) 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f6579cbe640 1 -- 172.21.15.28:0/1760500966 learned_addr learned my addr 172.21.15.28:0/1760500966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.106+0000 7f657a4bf640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6574103b80 0x7f657419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f657acc0640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6574109e10 0x7f657419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 -- 172.21.15.28:0/1760500966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6574103b80 msgr2=0x7f657419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6574103b80 0x7f657419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 -- 172.21.15.28:0/1760500966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6574109e10 msgr2=0x7f657419fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6574109e10 0x7f657419fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65741a0490 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f6579cbe640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f657419aac0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6568006010 tx=0x7f6568004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656803f030 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65741a0720 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6568003aa0 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568035070 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.107+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65741a6ee0 con 0x7f65741094c0 2024-09-21T05:45:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.108+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6574063b50 con 0x7f65741094c0 2024-09-21T05:45:19.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.110+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f6568042030 con 0x7f65741094c0 2024-09-21T05:45:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f65677fe640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65600b65f0 0x7f65600b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f657a4bf640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65600b65f0 0x7f65600b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6568107f40 con 0x7f65741094c0 2024-09-21T05:45:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f65677fe640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] conn(0x7f65600c0460 0x7f65600c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6568009080 con 0x7f65600c0460 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f65681082f0 con 0x7f65741094c0 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.111+0000 7f657acc0640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] conn(0x7f65600c0460 0x7f65600c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.112+0000 7f657a4bf640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65600b65f0 0x7f65600b8b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6570009bd0 tx=0x7f6570005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.112+0000 7f657acc0640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] conn(0x7f65600c0460 0x7f65600c2950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.113+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== osd.3 v2:172.21.15.28:6810/2300237806 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6568009080 con 0x7f65600c0460 2024-09-21T05:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.126+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 --> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6574063f50 con 0x7f65600c0460 2024-09-21T05:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.126+0000 7f65677fe640 1 -- 172.21.15.28:0/1760500966 <== osd.3 v2:172.21.15.28:6810/2300237806 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6574063f50 con 0x7f65600c0460 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] conn(0x7f65600c0460 msgr2=0x7f65600c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] conn(0x7f65600c0460 0x7f65600c2950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65600b65f0 msgr2=0x7f65600b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f65600b65f0 0x7f65600b8b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6570009bd0 tx=0x7f6570005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 msgr2=0x7f657419aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 --2- 172.21.15.28:0/1760500966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65741094c0 0x7f657419aac0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6568006010 tx=0x7f6568004060 comp rx=0 tx=0).stop 2024-09-21T05:45:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657acc0640 1 -- 172.21.15.28:0/1760500966 reap_dead start 2024-09-21T05:45:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 shutdown_connections 2024-09-21T05:45:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 >> 172.21.15.28:0/1760500966 conn(0x7f65740ff610 msgr2=0x7f6574101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 shutdown_connections 2024-09-21T05:45:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:19.127+0000 7f657c723640 1 -- 172.21.15.28:0/1760500966 wait complete. 2024-09-21T05:45:19.686 INFO:teuthology.orchestra.run.smithi028.stdout:128849018934 2024-09-21T05:45:19.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T05:45:20.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:20 smithi181 ceph-mon[34819]: pgmap v274: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 383 B/s wr, 10 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:20 smithi028 ceph-mon[31227]: pgmap v274: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 383 B/s wr, 10 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.556+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4196766963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108d10 msgr2=0x7f6d6c10f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.556+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4196766963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108d10 0x7f6d6c10f540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009ca0 tx=0x7f6d6402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4196766963 shutdown_connections 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4196766963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108d10 0x7f6d6c10f540 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4196766963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d6c1083c0 0x7f6d6c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4196766963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d6c102a80 0x7f6d6c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4196766963 >> 172.21.15.28:0/4196766963 conn(0x7f6d6c079740 msgr2=0x7f6d6c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4196766963 shutdown_connections 2024-09-21T05:45:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.557+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4196766963 wait complete. 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 Processor -- start 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 -- start start 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 0x7f6d6c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d6c1083c0 0x7f6d6c199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d6c108d10 0x7f6d6c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c112980 con 0x7f6d6c102a80 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c112800 con 0x7f6d6c108d10 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d70fe7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c112b00 con 0x7f6d6c1083c0 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d6a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 0x7f6d6c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d6a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 0x7f6d6c199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43996/0 (socket says 172.21.15.28:43996) 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.558+0000 7f6d6a59c640 1 -- 172.21.15.28:0/4173478883 learned_addr learned my addr 172.21.15.28:0/4173478883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6ad9d640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d6c108d10 0x7f6d6c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d69d9b640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d6c1083c0 0x7f6d6c199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 -- 172.21.15.28:0/4173478883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d6c1083c0 msgr2=0x7f6d6c199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d6c1083c0 0x7f6d6c199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 -- 172.21.15.28:0/4173478883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d6c108d10 msgr2=0x7f6d6c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d6c108d10 0x7f6d6c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c1a87b0 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d6a59c640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 0x7f6d6c199100 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00e9f0 tx=0x7f6d5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00cd80 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c1a8aa0 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d5c0054d0 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c010690 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.559+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c1a8fe0 con 0x7f6d6c102a80 2024-09-21T05:45:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.560+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d6c063ad0 con 0x7f6d6c102a80 2024-09-21T05:45:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.562+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f6d5c004e30 con 0x7f6d6c102a80 2024-09-21T05:45:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d5b7fe640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d540b65f0 0x7f6d540b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d69d9b640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d540b65f0 0x7f6d540b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6d5c0e7f60 con 0x7f6d6c102a80 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d5b7fe640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] conn(0x7f6d540c0460 0x7f6d540c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d5c015e80 con 0x7f6d540c0460 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f6d5c0e83b0 con 0x7f6d6c102a80 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d6ad9d640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] conn(0x7f6d540c0460 0x7f6d540c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.563+0000 7f6d69d9b640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d540b65f0 0x7f6d540b8b00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c105f60 tx=0x7f6d60005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.564+0000 7f6d6ad9d640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] conn(0x7f6d540c0460 0x7f6d540c2950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.565+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== osd.0 v2:172.21.15.28:6802/3566179718 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d5c015e80 con 0x7f6d540c0460 2024-09-21T05:45:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 --> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d6c063f50 con 0x7f6d540c0460 2024-09-21T05:45:20.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d5b7fe640 1 -- 172.21.15.28:0/4173478883 <== osd.0 v2:172.21.15.28:6802/3566179718 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6d6c063f50 con 0x7f6d540c0460 2024-09-21T05:45:20.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] conn(0x7f6d540c0460 msgr2=0x7f6d540c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] conn(0x7f6d540c0460 0x7f6d540c2950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d540b65f0 msgr2=0x7f6d540b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d540b65f0 0x7f6d540b8b00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c105f60 tx=0x7f6d60005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 msgr2=0x7f6d6c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 --2- 172.21.15.28:0/4173478883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c102a80 0x7f6d6c199100 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00e9f0 tx=0x7f6d5c00edf0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d6ad9d640 1 -- 172.21.15.28:0/4173478883 reap_dead start 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 shutdown_connections 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.581+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 >> 172.21.15.28:0/4173478883 conn(0x7f6d6c079740 msgr2=0x7f6d6c104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.582+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 shutdown_connections 2024-09-21T05:45:20.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.582+0000 7f6d70fe7640 1 -- 172.21.15.28:0/4173478883 wait complete. 2024-09-21T05:45:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:20 smithi129 ceph-mon[34885]: pgmap v274: 97 pgs: 2 peering, 95 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 383 B/s wr, 10 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:20.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/283355207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 msgr2=0x7f1a581098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/283355207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a581098c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009900 tx=0x7f1a4c02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/283355207 shutdown_connections 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/283355207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a58109e00 0x7f1a58110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/283355207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a581098c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/283355207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a58103bc0 0x7f1a58103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/283355207 >> 172.21.15.28:0/283355207 conn(0x7f1a5807b910 msgr2=0x7f1a5807bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/283355207 shutdown_connections 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.472+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/283355207 wait complete. 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 Processor -- start 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 -- start start 2024-09-21T05:45:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a58103bc0 0x7f1a5819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a5819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a58109e00 0x7f1a5819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58113ac0 con 0x7f1a581094b0 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58113940 con 0x7f1a58109e00 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a5dfb7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58113c40 con 0x7f1a58103bc0 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a5819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.473+0000 7f1a56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a5819aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43968/0 (socket says 172.21.15.28:43968) 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a57fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a58109e00 0x7f1a5819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 -- 172.21.15.28:0/4229822724 learned_addr learned my addr 172.21.15.28:0/4229822724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a577fe640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a58103bc0 0x7f1a5819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 -- 172.21.15.28:0/4229822724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a58103bc0 msgr2=0x7f1a5819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a58103bc0 0x7f1a5819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 -- 172.21.15.28:0/4229822724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a58109e00 msgr2=0x7f1a5819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a58109e00 0x7f1a5819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a581a0380 con 0x7f1a581094b0 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a56ffd640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a5819aa70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c02a930 tx=0x7f1a4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c03f030 con 0x7f1a581094b0 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.474+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a581a0610 con 0x7f1a581094b0 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.475+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4c003cb0 con 0x7f1a581094b0 2024-09-21T05:45:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.475+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c035070 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.475+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a581a0b40 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.475+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a58063b50 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.477+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f1a4c042030 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a54ff9640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1a400b6620 0x7f1a400b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a577fe640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1a400b6620 0x7f1a400b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1a4c107fe0 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a54ff9640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] conn(0x7f1a400c0490 0x7f1a400c2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a4c03dec0 con 0x7f1a400c0490 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f1a4c108410 con 0x7f1a581094b0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a57fff640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] conn(0x7f1a400c0490 0x7f1a400c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.478+0000 7f1a577fe640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1a400b6620 0x7f1a400b8b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1a480058c0 tx=0x7f1a4800a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.479+0000 7f1a57fff640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] conn(0x7f1a400c0490 0x7f1a400c2980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.479+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== osd.6 v2:172.21.15.28:6818/88277262 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a4c03dec0 con 0x7f1a400c0490 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.493+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 --> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1a58063f50 con 0x7f1a400c0490 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a54ff9640 1 -- 172.21.15.28:0/4229822724 <== osd.6 v2:172.21.15.28:6818/88277262 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1a58063f50 con 0x7f1a400c0490 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] conn(0x7f1a400c0490 msgr2=0x7f1a400c2980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] conn(0x7f1a400c0490 0x7f1a400c2980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1a400b6620 msgr2=0x7f1a400b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1a400b6620 0x7f1a400b8b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1a480058c0 tx=0x7f1a4800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 msgr2=0x7f1a5819aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 --2- 172.21.15.28:0/4229822724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a581094b0 0x7f1a5819aa70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c02a930 tx=0x7f1a4c004270 comp rx=0 tx=0).stop 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a57fff640 1 -- 172.21.15.28:0/4229822724 reap_dead start 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 shutdown_connections 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 >> 172.21.15.28:0/4229822724 conn(0x7f1a5807b910 msgr2=0x7f1a58105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 shutdown_connections 2024-09-21T05:45:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.494+0000 7f1a5dfb7640 1 -- 172.21.15.28:0/4229822724 wait complete. 2024-09-21T05:45:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 -- 172.21.15.28:0/3580989271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 msgr2=0x7f068010d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 --2- 172.21.15.28:0/3580989271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068010d850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f066c009900 tx=0x7f066c02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 -- 172.21.15.28:0/3580989271 shutdown_connections 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 --2- 172.21.15.28:0/3580989271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f068010de10 0x7f0680110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 --2- 172.21.15.28:0/3580989271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068010d850 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 --2- 172.21.15.28:0/3580989271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06801049b0 0x7f0680104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 -- 172.21.15.28:0/3580989271 >> 172.21.15.28:0/3580989271 conn(0x7f0680079730 msgr2=0x7f0680079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 -- 172.21.15.28:0/3580989271 shutdown_connections 2024-09-21T05:45:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.441+0000 7f0686cea640 1 -- 172.21.15.28:0/3580989271 wait complete. 2024-09-21T05:45:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.442+0000 7f0686cea640 1 Processor -- start 2024-09-21T05:45:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.442+0000 7f0686cea640 1 -- start start 2024-09-21T05:45:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06801049b0 0x7f068019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f068010de10 0x7f068019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0680113a70 con 0x7f0680105300 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06801138f0 con 0x7f06801049b0 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0686cea640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0680113bf0 con 0x7f068010de10 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44012/0 (socket says 172.21.15.28:44012) 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 -- 172.21.15.28:0/4012553586 learned_addr learned my addr 172.21.15.28:0/4012553586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0684a86640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06801049b0 0x7f068019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f0685287640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f068010de10 0x7f068019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 -- 172.21.15.28:0/4012553586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f068010de10 msgr2=0x7f068019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f068010de10 0x7f068019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 -- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06801049b0 msgr2=0x7f068019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06801049b0 0x7f068019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.443+0000 7f067ffff640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06801a0430 con 0x7f0680105300 2024-09-21T05:45:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f067ffff640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068019aa60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f066c02a930 tx=0x7f066c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c03f030 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066c003aa0 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06801a06c0 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c035070 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.444+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06801a6e80 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.445+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0680063ad0 con 0x7f0680105300 2024-09-21T05:45:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.447+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f066c042030 con 0x7f0680105300 2024-09-21T05:45:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.447+0000 7f067dffb640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06700b65f0 0x7f06700b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f0684a86640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06700b65f0 0x7f06700b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f066c1080b0 con 0x7f0680105300 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f067dffb640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] conn(0x7f06700c0460 0x7f06700c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f066c044e50 con 0x7f06700c0460 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f066c108590 con 0x7f0680105300 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f0685287640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] conn(0x7f06700c0460 0x7f06700c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f0684a86640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06700b65f0 0x7f06700b8b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f068006bc60 tx=0x7f067400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.448+0000 7f0685287640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] conn(0x7f06700c0460 0x7f06700c2950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.449+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== osd.1 v2:172.21.15.129:6800/1623835479 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f066c044e50 con 0x7f06700c0460 2024-09-21T05:45:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.463+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 --> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0680063ed0 con 0x7f06700c0460 2024-09-21T05:45:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.464+0000 7f067dffb640 1 -- 172.21.15.28:0/4012553586 <== osd.1 v2:172.21.15.129:6800/1623835479 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0680063ed0 con 0x7f06700c0460 2024-09-21T05:45:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.464+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] conn(0x7f06700c0460 msgr2=0x7f06700c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.464+0000 7f0686cea640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] conn(0x7f06700c0460 0x7f06700c2950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06700b65f0 msgr2=0x7f06700b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06700b65f0 0x7f06700b8b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f068006bc60 tx=0x7f067400b390 comp rx=0 tx=0).stop 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 msgr2=0x7f068019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 --2- 172.21.15.28:0/4012553586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0680105300 0x7f068019aa60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f066c02a930 tx=0x7f066c004270 comp rx=0 tx=0).stop 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0685287640 1 -- 172.21.15.28:0/4012553586 reap_dead start 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 shutdown_connections 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 >> 172.21.15.28:0/4012553586 conn(0x7f0680079730 msgr2=0x7f06801008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 shutdown_connections 2024-09-21T05:45:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.465+0000 7f0686cea640 1 -- 172.21.15.28:0/4012553586 wait complete. 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 -- 172.21.15.28:0/267784001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a8102a60 msgr2=0x7f79a8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/267784001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a8102a60 0x7f79a8102e70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7998009900 tx=0x7f799802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 -- 172.21.15.28:0/267784001 shutdown_connections 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/267784001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79a8108cf0 0x7f79a810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/267784001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a81083a0 0x7f79a81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/267784001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a8102a60 0x7f79a8102e70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 -- 172.21.15.28:0/267784001 >> 172.21.15.28:0/267784001 conn(0x7f79a8079730 msgr2=0x7f79a8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 -- 172.21.15.28:0/267784001 shutdown_connections 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.840+0000 7f79ac9e6640 1 -- 172.21.15.28:0/267784001 wait complete. 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.841+0000 7f79ac9e6640 1 Processor -- start 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.841+0000 7f79ac9e6640 1 -- start start 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a8102a60 0x7f79a8199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 0x7f79a81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79a8108cf0 0x7f79a819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8112960 con 0x7f79a81083a0 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79a81127e0 con 0x7f79a8102a60 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79ac9e6640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8112ae0 con 0x7f79a8108cf0 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 0x7f79a81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 0x7f79a81998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44030/0 (socket says 172.21.15.28:44030) 2024-09-21T05:45:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 -- 172.21.15.28:0/3957961531 learned_addr learned my addr 172.21.15.28:0/3957961531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a6d9d640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79a8108cf0 0x7f79a819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a659c640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a8102a60 0x7f79a8199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 -- 172.21.15.28:0/3957961531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79a8108cf0 msgr2=0x7f79a819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79a8108cf0 0x7f79a819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 -- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a8102a60 msgr2=0x7f79a8199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a8102a60 0x7f79a8199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a819f2a0 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a659c640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79a8102a60 0x7f79a8199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.842+0000 7f79a5d9b640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 0x7f79a81998d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f799c00e9c0 tx=0x7f799c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.843+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c00cd80 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.843+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f799c0054d0 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.843+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a819f590 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.843+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c010430 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.843+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a806ae60 con 0x7f79a81083a0 2024-09-21T05:45:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.844+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79a806b430 con 0x7f79a81083a0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.846+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f799c004f90 con 0x7f79a81083a0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.846+0000 7f79977fe640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79900b65f0 0x7f79900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.846+0000 7f79a659c640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79900b65f0 0x7f79900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f799c0e7f50 con 0x7f79a81083a0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79977fe640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] conn(0x7f79900c0460 0x7f79900c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f799c011e50 con 0x7f79900c0460 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f799c0e8300 con 0x7f79a81083a0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79a659c640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79900b65f0 0x7f79900b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7998031440 tx=0x7f7998009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79a6d9d640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] conn(0x7f79900c0460 0x7f79900c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.847+0000 7f79a6d9d640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] conn(0x7f79900c0460 0x7f79900c2950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.848+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== osd.10 v2:172.21.15.129:6824/3703525783 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f799c011e50 con 0x7f79900c0460 2024-09-21T05:45:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.862+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 --> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f79a806b8b0 con 0x7f79900c0460 2024-09-21T05:45:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.862+0000 7f79977fe640 1 -- 172.21.15.28:0/3957961531 <== osd.10 v2:172.21.15.129:6824/3703525783 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f79a806b8b0 con 0x7f79900c0460 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] conn(0x7f79900c0460 msgr2=0x7f79900c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] conn(0x7f79900c0460 0x7f79900c2950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79900b65f0 msgr2=0x7f79900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79900b65f0 0x7f79900b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7998031440 tx=0x7f7998009680 comp rx=0 tx=0).stop 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 msgr2=0x7f79a81998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 --2- 172.21.15.28:0/3957961531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a81083a0 0x7f79a81998d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f799c00e9c0 tx=0x7f799c00ee80 comp rx=0 tx=0).stop 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79a6d9d640 1 -- 172.21.15.28:0/3957961531 reap_dead start 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 shutdown_connections 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 >> 172.21.15.28:0/3957961531 conn(0x7f79a8079730 msgr2=0x7f79a8101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 shutdown_connections 2024-09-21T05:45:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:21.863+0000 7f79ac9e6640 1 -- 172.21.15.28:0/3957961531 wait complete. 2024-09-21T05:45:22.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:22 smithi181 ceph-mon[34819]: pgmap v275: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:22.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:22 smithi028 ceph-mon[31227]: pgmap v275: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:22 smithi129 ceph-mon[34885]: pgmap v275: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s; 164 B/s, 1 objects/s recovering 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.747+0000 7f3bd383b640 1 -- 172.21.15.28:0/1818518736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc1025d0 msgr2=0x7f3bcc1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.747+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1818518736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc1025d0 0x7f3bcc1029e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009900 tx=0x7f3bc802a480 comp rx=0 tx=0).stop 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 -- 172.21.15.28:0/1818518736 shutdown_connections 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1818518736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bcc110180 0x7f3bcc1125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1818518736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 0x7f3bcc10fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1818518736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc1025d0 0x7f3bcc1029e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 -- 172.21.15.28:0/1818518736 >> 172.21.15.28:0/1818518736 conn(0x7f3bcc079730 msgr2=0x7f3bcc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 -- 172.21.15.28:0/1818518736 shutdown_connections 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 -- 172.21.15.28:0/1818518736 wait complete. 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.748+0000 7f3bd383b640 1 Processor -- start 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 -- start start 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bcc1025d0 0x7f3bcc19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 0x7f3bcc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 0x7f3bcc1a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc115c70 con 0x7f3bcc110180 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc115af0 con 0x7f3bcc102f20 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd383b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc115df0 con 0x7f3bcc1025d0 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 0x7f3bcc1a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 0x7f3bcc1a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44046/0 (socket says 172.21.15.28:44046) 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 -- 172.21.15.28:0/1358362157 learned_addr learned my addr 172.21.15.28:0/1358362157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd15d7640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bcc1025d0 0x7f3bcc19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd0dd6640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 0x7f3bcc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 -- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bcc1025d0 msgr2=0x7f3bcc19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bcc1025d0 0x7f3bcc19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 -- 172.21.15.28:0/1358362157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 msgr2=0x7f3bcc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 0x7f3bcc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd1dd8640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc1a2550 con 0x7f3bcc110180 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.749+0000 7f3bd0dd6640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bcc102f20 0x7f3bcc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bd1dd8640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 0x7f3bcc1a1ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400cce0 tx=0x7f3bc400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400f450 con 0x7f3bcc110180 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc400faa0 con 0x7f3bcc110180 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4018b70 con 0x7f3bcc110180 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc1a2840 con 0x7f3bcc110180 2024-09-21T05:45:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.750+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1a9060 con 0x7f3bcc110180 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.751+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bcc063b50 con 0x7f3bcc110180 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.754+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f3bc4018d10 con 0x7f3bcc110180 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.754+0000 7f3bc27fc640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bb80b66a0 0x7f3bb80b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.754+0000 7f3bd15d7640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bb80b66a0 0x7f3bb80b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3bc40e7d70 con 0x7f3bcc110180 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bc27fc640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] conn(0x7f3bb80c0510 0x7f3bb80c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bb80c30d0 con 0x7f3bb80c0510 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f3bc40e81c0 con 0x7f3bcc110180 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bd15d7640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bb80b66a0 0x7f3bb80b8bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8031440 tx=0x7f3bc8009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.755+0000 7f3bd0dd6640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] conn(0x7f3bb80c0510 0x7f3bb80c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.756+0000 7f3bd0dd6640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] conn(0x7f3bb80c0510 0x7f3bb80c2a00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.756+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== osd.2 v2:172.21.15.181:6800/3260464300 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bb80c30d0 con 0x7f3bb80c0510 2024-09-21T05:45:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.771+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 --> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bcc063fd0 con 0x7f3bb80c0510 2024-09-21T05:45:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.771+0000 7f3bc27fc640 1 -- 172.21.15.28:0/1358362157 <== osd.2 v2:172.21.15.181:6800/3260464300 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3bcc063fd0 con 0x7f3bb80c0510 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] conn(0x7f3bb80c0510 msgr2=0x7f3bb80c2a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] conn(0x7f3bb80c0510 0x7f3bb80c2a00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bb80b66a0 msgr2=0x7f3bb80b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3bb80b66a0 0x7f3bb80b8bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8031440 tx=0x7f3bc8009560 comp rx=0 tx=0).stop 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 msgr2=0x7f3bcc1a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 --2- 172.21.15.28:0/1358362157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bcc110180 0x7f3bcc1a1ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400cce0 tx=0x7f3bc400a620 comp rx=0 tx=0).stop 2024-09-21T05:45:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd1dd8640 1 -- 172.21.15.28:0/1358362157 reap_dead start 2024-09-21T05:45:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 shutdown_connections 2024-09-21T05:45:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 >> 172.21.15.28:0/1358362157 conn(0x7f3bcc079730 msgr2=0x7f3bcc1115c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 shutdown_connections 2024-09-21T05:45:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:22.772+0000 7f3bd383b640 1 -- 172.21.15.28:0/1358362157 wait complete. 2024-09-21T05:45:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 -- 172.21.15.28:0/1089554256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 msgr2=0x7f0d10110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 --2- 172.21.15.28:0/1089554256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d10110620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009ca0 tx=0x7f0d0802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 -- 172.21.15.28:0/1089554256 shutdown_connections 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 --2- 172.21.15.28:0/1089554256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d10110620 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 --2- 172.21.15.28:0/1089554256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d101094a0 0x7f0d101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 --2- 172.21.15.28:0/1089554256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d10103b60 0x7f0d10103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 -- 172.21.15.28:0/1089554256 >> 172.21.15.28:0/1089554256 conn(0x7f0d1007b850 msgr2=0x7f0d1007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 -- 172.21.15.28:0/1089554256 shutdown_connections 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 -- 172.21.15.28:0/1089554256 wait complete. 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.060+0000 7f0d16c11640 1 Processor -- start 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 -- start start 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d10103b60 0x7f0d1019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d101094a0 0x7f0d1019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d1019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d10113ab0 con 0x7f0d10109df0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d10113930 con 0x7f0d10103b60 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d16c11640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d10113c30 con 0x7f0d101094a0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d1019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d1019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43958/0 (socket says 172.21.15.28:43958) 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 -- 172.21.15.28:0/3932000749 learned_addr learned my addr 172.21.15.28:0/3932000749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d0ffff640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d101094a0 0x7f0d1019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d149ad640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d10103b60 0x7f0d1019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 -- 172.21.15.28:0/3932000749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d101094a0 msgr2=0x7f0d1019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d101094a0 0x7f0d1019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 -- 172.21.15.28:0/3932000749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d10103b60 msgr2=0x7f0d1019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d10103b60 0x7f0d1019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.061+0000 7f0d151ae640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d101a03b0 con 0x7f0d10109df0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d151ae640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d1019fc80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d08005cc0 tx=0x7f0d08031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003b70 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d101a0640 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d080318f0 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0802f950 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.062+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d101a6e00 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.063+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d10063b50 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.065+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f0d0802fb10 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.065+0000 7f0d0dffb640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d000b65f0 0x7f0d000b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.065+0000 7f0d149ad640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d000b65f0 0x7f0d000b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.065+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0d081089a0 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d0dffb640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] conn(0x7f0d000c0460 0x7f0d000c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d08045e50 con 0x7f0d000c0460 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f0d08108df0 con 0x7f0d10109df0 2024-09-21T05:45:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d0ffff640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] conn(0x7f0d000c0460 0x7f0d000c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d149ad640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d000b65f0 0x7f0d000b8b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009bd0 tx=0x7f0d04005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.066+0000 7f0d0ffff640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] conn(0x7f0d000c0460 0x7f0d000c2950 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.067+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== osd.9 v2:172.21.15.28:6826/2894928742 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d08045e50 con 0x7f0d000c0460 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.080+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 --> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0d10063f50 con 0x7f0d000c0460 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d0dffb640 1 -- 172.21.15.28:0/3932000749 <== osd.9 v2:172.21.15.28:6826/2894928742 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0d10063f50 con 0x7f0d000c0460 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] conn(0x7f0d000c0460 msgr2=0x7f0d000c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] conn(0x7f0d000c0460 0x7f0d000c2950 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d000b65f0 msgr2=0x7f0d000b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0d000b65f0 0x7f0d000b8b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009bd0 tx=0x7f0d04005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 msgr2=0x7f0d1019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 --2- 172.21.15.28:0/3932000749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d10109df0 0x7f0d1019fc80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d08005cc0 tx=0x7f0d08031700 comp rx=0 tx=0).stop 2024-09-21T05:45:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d151ae640 1 -- 172.21.15.28:0/3932000749 reap_dead start 2024-09-21T05:45:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 shutdown_connections 2024-09-21T05:45:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 >> 172.21.15.28:0/3932000749 conn(0x7f0d1007b850 msgr2=0x7f0d101074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 shutdown_connections 2024-09-21T05:45:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:20.081+0000 7f0d16c11640 1 -- 172.21.15.28:0/3932000749 wait complete. 2024-09-21T05:45:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:236223201311 2024-09-21T05:45:23.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.9 2024-09-21T05:45:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- 172.21.15.28:0/200869747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41083b0 msgr2=0x7fd7a41087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 --2- 172.21.15.28:0/200869747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41083b0 0x7fd7a41087c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd794009900 tx=0x7fd79402a410 comp rx=0 tx=0).stop 2024-09-21T05:45:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- 172.21.15.28:0/200869747 shutdown_connections 2024-09-21T05:45:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 --2- 172.21.15.28:0/200869747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7a4108d00 0x7fd7a410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 --2- 172.21.15.28:0/200869747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a41083b0 0x7fd7a41087c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 --2- 172.21.15.28:0/200869747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a4102ac0 0x7fd7a4102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- 172.21.15.28:0/200869747 >> 172.21.15.28:0/200869747 conn(0x7fd7a4079790 msgr2=0x7fd7a4079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- 172.21.15.28:0/200869747 shutdown_connections 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- 172.21.15.28:0/200869747 wait complete. 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 Processor -- start 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.874+0000 7fd7abf78640 1 -- start start 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a4102ac0 0x7fd7a4110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7a41083b0 0x7fd7a4110550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 0x7fd7a4110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4113ae0 con 0x7fd7a4108d00 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4113960 con 0x7fd7a4102ac0 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7abf78640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4113c60 con 0x7fd7a41083b0 2024-09-21T05:45:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 0x7fd7a4110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 0x7fd7a4110a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44062/0 (socket says 172.21.15.28:44062) 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 -- 172.21.15.28:0/1567942339 learned_addr learned my addr 172.21.15.28:0/1567942339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7a9513640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7a41083b0 0x7fd7a4110550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7a9d14640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a4102ac0 0x7fd7a4110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 -- 172.21.15.28:0/1567942339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7a41083b0 msgr2=0x7fd7a4110550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7a41083b0 0x7fd7a4110550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 -- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a4102ac0 msgr2=0x7fd7a4110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a4102ac0 0x7fd7a4110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.875+0000 7fd7aa515640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a41a8790 con 0x7fd7a4108d00 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd7aa515640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 0x7fd7a4110a90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009fd0 tx=0x7fd79c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c017070 con 0x7fd7a4108d00 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a41a8a80 con 0x7fd7a4108d00 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79c003e20 con 0x7fd7a4108d00 2024-09-21T05:45:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c010460 con 0x7fd7a4108d00 2024-09-21T05:45:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.876+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a8f40 con 0x7fd7a4108d00 2024-09-21T05:45:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.877+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7a4063b50 con 0x7fd7a4108d00 2024-09-21T05:45:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.879+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fd79c003970 con 0x7fd7a4108d00 2024-09-21T05:45:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.879+0000 7fd79affd640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd7900b65f0 0x7fd7900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd7a9d14640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd7900b65f0 0x7fd7900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd79c0e76d0 con 0x7fd7a4108d00 2024-09-21T05:45:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd79affd640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] conn(0x7fd7900c0460 0x7fd7900c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd79c021e50 con 0x7fd7900c0460 2024-09-21T05:45:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fd79c0e7b20 con 0x7fd7a4108d00 2024-09-21T05:45:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd7a9d14640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd7900b65f0 0x7fd7900b8b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4111a60 tx=0x7fd7a000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.880+0000 7fd7a9513640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] conn(0x7fd7900c0460 0x7fd7900c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.881+0000 7fd7a9513640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] conn(0x7fd7900c0460 0x7fd7900c2950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.882+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== osd.7 v2:172.21.15.129:6816/317230964 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd79c021e50 con 0x7fd7900c0460 2024-09-21T05:45:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.896+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 --> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7a4064000 con 0x7fd7900c0460 2024-09-21T05:45:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.896+0000 7fd79affd640 1 -- 172.21.15.28:0/1567942339 <== osd.7 v2:172.21.15.129:6816/317230964 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd7a4064000 con 0x7fd7900c0460 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] conn(0x7fd7900c0460 msgr2=0x7fd7900c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] conn(0x7fd7900c0460 0x7fd7900c2950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd7900b65f0 msgr2=0x7fd7900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd7900b65f0 0x7fd7900b8b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4111a60 tx=0x7fd7a000b410 comp rx=0 tx=0).stop 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 msgr2=0x7fd7a4110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 --2- 172.21.15.28:0/1567942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a4108d00 0x7fd7a4110a90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009fd0 tx=0x7fd79c00b260 comp rx=0 tx=0).stop 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7aa515640 1 -- 172.21.15.28:0/1567942339 reap_dead start 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 shutdown_connections 2024-09-21T05:45:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 >> 172.21.15.28:0/1567942339 conn(0x7fd7a4079790 msgr2=0x7fd7a4104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:23.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 shutdown_connections 2024-09-21T05:45:23.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:23.897+0000 7fd7abf78640 1 -- 172.21.15.28:0/1567942339 wait complete. 2024-09-21T05:45:24.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:24 smithi181 ceph-mon[34819]: pgmap v276: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 138 B/s, 1 objects/s recovering 2024-09-21T05:45:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:24 smithi028 ceph-mon[31227]: pgmap v276: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 138 B/s, 1 objects/s recovering 2024-09-21T05:45:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:24 smithi129 ceph-mon[34885]: pgmap v276: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 138 B/s, 1 objects/s recovering 2024-09-21T05:45:24.680 INFO:teuthology.orchestra.run.smithi028.stdout:38654705737 2024-09-21T05:45:24.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T05:45:25.087 INFO:teuthology.orchestra.run.smithi028.stdout:167503724589 2024-09-21T05:45:25.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T05:45:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/2252608172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58108400 msgr2=0x7f5a58108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/2252608172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58108400 0x7f5a58108810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009900 tx=0x7f5a4c02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/2252608172 shutdown_connections 2024-09-21T05:45:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/2252608172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a58108d50 0x7f5a5810f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/2252608172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58108400 0x7f5a58108810 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/2252608172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a58103190 0x7f5a581035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/2252608172 >> 172.21.15.28:0/2252608172 conn(0x7f5a58079f90 msgr2=0x7f5a5807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/2252608172 shutdown_connections 2024-09-21T05:45:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.327+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/2252608172 wait complete. 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.328+0000 7f5a5cc3a640 1 Processor -- start 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.328+0000 7f5a5cc3a640 1 -- start start 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 0x7f5a580fedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a58108400 0x7f5a580ff330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a58108d50 0x7f5a580ff870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a58112a10 con 0x7f5a58103190 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a58112890 con 0x7f5a58108400 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5cc3a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5a58112b90 con 0x7f5a58108d50 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 0x7f5a580fedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 0x7f5a580fedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56216/0 (socket says 172.21.15.28:56216) 2024-09-21T05:45:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 -- 172.21.15.28:0/1668611897 learned_addr learned my addr 172.21.15.28:0/1668611897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a56d9d640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a58108d50 0x7f5a580ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a55d9b640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a58108400 0x7f5a580ff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 -- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a58108d50 msgr2=0x7f5a580ff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a58108d50 0x7f5a580ff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 -- 172.21.15.28:0/1668611897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a58108400 msgr2=0x7f5a580ff330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a58108400 0x7f5a580ff330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a581a8720 con 0x7f5a58103190 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a5659c640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 0x7f5a580fedf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800d950 tx=0x7f5a4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.329+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48017070 con 0x7f5a58103190 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.330+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a581a8a10 con 0x7f5a58103190 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.330+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a48009c30 con 0x7f5a58103190 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.330+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48020060 con 0x7f5a58103190 2024-09-21T05:45:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.330+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a581a8eb0 con 0x7f5a58103190 2024-09-21T05:45:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.331+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a58063b50 con 0x7f5a58103190 2024-09-21T05:45:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.332+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f5a48010460 con 0x7f5a58103190 2024-09-21T05:45:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.333+0000 7f5a477fe640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5a400b65f0 0x7f5a400b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.333+0000 7f5a55d9b640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5a400b65f0 0x7f5a400b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.333+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5a480e71b0 con 0x7f5a58103190 2024-09-21T05:45:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.333+0000 7f5a477fe640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] conn(0x7f5a400c0460 0x7f5a400c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.334+0000 7f5a55d9b640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5a400b65f0 0x7f5a400b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009230 tx=0x7f5a4c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.334+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a48021e50 con 0x7f5a400c0460 2024-09-21T05:45:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.334+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f5a480e7600 con 0x7f5a58103190 2024-09-21T05:45:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.334+0000 7f5a56d9d640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] conn(0x7f5a400c0460 0x7f5a400c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.334+0000 7f5a56d9d640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] conn(0x7f5a400c0460 0x7f5a400c2950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.336+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== osd.8 v2:172.21.15.181:6816/3198301680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a48021e50 con 0x7f5a400c0460 2024-09-21T05:45:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.349+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 --> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a58063f50 con 0x7f5a400c0460 2024-09-21T05:45:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a477fe640 1 -- 172.21.15.28:0/1668611897 <== osd.8 v2:172.21.15.181:6816/3198301680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5a58063f50 con 0x7f5a400c0460 2024-09-21T05:45:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] conn(0x7f5a400c0460 msgr2=0x7f5a400c2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] conn(0x7f5a400c0460 0x7f5a400c2950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5a400b65f0 msgr2=0x7f5a400b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5a400b65f0 0x7f5a400b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009230 tx=0x7f5a4c031400 comp rx=0 tx=0).stop 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 msgr2=0x7f5a580fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 --2- 172.21.15.28:0/1668611897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a58103190 0x7f5a580fedf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800d950 tx=0x7f5a4800de10 comp rx=0 tx=0).stop 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a56d9d640 1 -- 172.21.15.28:0/1668611897 reap_dead start 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 shutdown_connections 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.350+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 >> 172.21.15.28:0/1668611897 conn(0x7f5a58079f90 msgr2=0x7f5a58101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.351+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 shutdown_connections 2024-09-21T05:45:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:25.351+0000 7f5a5cc3a640 1 -- 172.21.15.28:0/1668611897 wait complete. 2024-09-21T05:45:25.990 INFO:teuthology.orchestra.run.smithi028.stdout:77309411394 2024-09-21T05:45:25.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T05:45:26.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:26 smithi181 ceph-mon[34819]: pgmap v277: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 125 B/s, 1 objects/s recovering 2024-09-21T05:45:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:26 smithi028 ceph-mon[31227]: pgmap v277: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 125 B/s, 1 objects/s recovering 2024-09-21T05:45:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:26 smithi129 ceph-mon[34885]: pgmap v277: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 125 B/s, 1 objects/s recovering 2024-09-21T05:45:26.673 INFO:teuthology.orchestra.run.smithi028.stdout:214748364836 2024-09-21T05:45:26.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.8 2024-09-21T05:45:27.075 INFO:teuthology.orchestra.run.smithi028.stdout:188978561065 2024-09-21T05:45:27.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T05:45:28.074 INFO:teuthology.orchestra.run.smithi028.stdout:60129542215 2024-09-21T05:45:28.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T05:45:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:28 smithi028 ceph-mon[31227]: pgmap v278: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:28.518 INFO:teuthology.orchestra.run.smithi028.stdout:257698037786 2024-09-21T05:45:28.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.10 2024-09-21T05:45:28.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:28 smithi181 ceph-mon[34819]: pgmap v278: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:28 smithi129 ceph-mon[34885]: pgmap v278: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:103079215163 2024-09-21T05:45:28.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T05:45:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2272055416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101f70 msgr2=0x7fc058102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2272055416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101f70 0x7fc058102380 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc048009900 tx=0x7fc04802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2272055416 shutdown_connections 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2272055416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058109e00 0x7fc058110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2272055416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0581094b0 0x7fc0581098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2272055416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101f70 0x7fc058102380 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.124+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2272055416 >> 172.21.15.28:0/2272055416 conn(0x7fc058077c60 msgr2=0x7fc0580780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.125+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2272055416 shutdown_connections 2024-09-21T05:45:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.125+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2272055416 wait complete. 2024-09-21T05:45:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.126+0000 7fc05d5a7640 1 Processor -- start 2024-09-21T05:45:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.127+0000 7fc05d5a7640 1 -- start start 2024-09-21T05:45:29.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.127+0000 7fc05d5a7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 0x7fc05819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.127+0000 7fc05d5a7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0581094b0 0x7fc05819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc05d5a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058109e00 0x7fc05819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 0x7fc05819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 0x7fc05819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33182/0 (socket says 172.21.15.28:33182) 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc056ffd640 1 -- 172.21.15.28:0/2400926064 learned_addr learned my addr 172.21.15.28:0/2400926064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc058113ac0 con 0x7fc058109e00 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058109e00 0x7fc05819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc058113940 con 0x7fc058101f70 2024-09-21T05:45:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.128+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc058113c40 con 0x7fc0581094b0 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0581094b0 0x7fc05819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 -- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0581094b0 msgr2=0x7fc05819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0581094b0 0x7fc05819a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 -- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058109e00 msgr2=0x7fc05819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058109e00 0x7fc05819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0581a03c0 con 0x7fc058101f70 2024-09-21T05:45:29.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058109e00 0x7fc05819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:29.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.129+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 0x7fc05819a450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc048005940 tx=0x7fc0480040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:29.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.130+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04803f030 con 0x7fc058101f70 2024-09-21T05:45:29.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.130+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0480044c0 con 0x7fc058101f70 2024-09-21T05:45:29.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.130+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048035070 con 0x7fc058101f70 2024-09-21T05:45:29.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.130+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0581a0650 con 0x7fc058101f70 2024-09-21T05:45:29.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.130+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0581a0a20 con 0x7fc058101f70 2024-09-21T05:45:29.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.132+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc058063b10 con 0x7fc058101f70 2024-09-21T05:45:29.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.134+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fc048042030 con 0x7fc058101f70 2024-09-21T05:45:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc03ffff640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b66f0 0x7fc0400b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc048108160 con 0x7fc058101f70 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b66f0 0x7fc0400b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc03ffff640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] conn(0x7fc0400c0560 0x7fc0400c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0400c3120 con 0x7fc0400c0560 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.135+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fc048108550 con 0x7fc058101f70 2024-09-21T05:45:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.136+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b66f0 0x7fc0400b8c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc0581091f0 tx=0x7fc04c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.138+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] conn(0x7fc0400c0560 0x7fc0400c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.141+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] conn(0x7fc0400c0560 0x7fc0400c2a50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.142+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== osd.11 v2:172.21.15.181:6824/3814712780 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0400c3120 con 0x7fc0400c0560 2024-09-21T05:45:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.163+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 --> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc058063fe0 con 0x7fc0400c0560 2024-09-21T05:45:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc03ffff640 1 -- 172.21.15.28:0/2400926064 <== osd.11 v2:172.21.15.181:6824/3814712780 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc058063fe0 con 0x7fc0400c0560 2024-09-21T05:45:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] conn(0x7fc0400c0560 msgr2=0x7fc0400c2a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] conn(0x7fc0400c0560 0x7fc0400c2a50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b66f0 msgr2=0x7fc0400b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b66f0 0x7fc0400b8c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc0581091f0 tx=0x7fc04c005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 msgr2=0x7fc05819a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc05d5a7640 1 --2- 172.21.15.28:0/2400926064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058101f70 0x7fc05819a450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc048005940 tx=0x7fc0480040c0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.164+0000 7fc0577fe640 1 -- 172.21.15.28:0/2400926064 reap_dead start 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.165+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 shutdown_connections 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.165+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 >> 172.21.15.28:0/2400926064 conn(0x7fc058077c60 msgr2=0x7fc058107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.165+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 shutdown_connections 2024-09-21T05:45:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.165+0000 7fc05d5a7640 1 -- 172.21.15.28:0/2400926064 wait complete. 2024-09-21T05:45:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.706 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 -- 172.21.15.28:0/3900482912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 msgr2=0x7ff4901098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 --2- 172.21.15.28:0/3900482912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff4901098c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff480009900 tx=0x7ff48002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 -- 172.21.15.28:0/3900482912 shutdown_connections 2024-09-21T05:45:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 --2- 172.21.15.28:0/3900482912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 0x7ff490110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 --2- 172.21.15.28:0/3900482912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff4901098c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 --2- 172.21.15.28:0/3900482912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 0x7ff490103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 -- 172.21.15.28:0/3900482912 >> 172.21.15.28:0/3900482912 conn(0x7ff4900ff5e0 msgr2=0x7ff490101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 -- 172.21.15.28:0/3900482912 shutdown_connections 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.908+0000 7ff494da5640 1 -- 172.21.15.28:0/3900482912 wait complete. 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 Processor -- start 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 -- start start 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 0x7ff49019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 0x7ff49019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff490113ac0 con 0x7ff4901094b0 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff490113940 con 0x7ff490109e00 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.909+0000 7ff494da5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff490113c40 con 0x7ff490103b70 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56248/0 (socket says 172.21.15.28:56248) 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 -- 172.21.15.28:0/223126992 learned_addr learned my addr 172.21.15.28:0/223126992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48ed9d640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 0x7ff49019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48e59c640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 0x7ff49019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 -- 172.21.15.28:0/223126992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 msgr2=0x7ff49019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 0x7ff49019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 -- 172.21.15.28:0/223126992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 msgr2=0x7ff49019fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 0x7ff49019fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 -- 172.21.15.28:0/223126992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4901a0480 con 0x7ff4901094b0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff48dd9b640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff48002a930 tx=0x7ff480004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48003f030 con 0x7ff4901094b0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff480003aa0 con 0x7ff4901094b0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4901a0710 con 0x7ff4901094b0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.910+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480035070 con 0x7ff4901094b0 2024-09-21T05:45:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.911+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4901a6ed0 con 0x7ff4901094b0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.912+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff490103f80 con 0x7ff4901094b0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.913+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7ff480042030 con 0x7ff4901094b0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.914+0000 7ff4877fe640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 0x7ff47c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.914+0000 7ff48e59c640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 0x7ff47c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.914+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff480107c10 con 0x7ff4901094b0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.915+0000 7ff48e59c640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 0x7ff47c0b8b80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff478005fd0 tx=0x7ff47800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:29.919+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff4800d2620 con 0x7ff4901094b0 2024-09-21T05:45:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:30.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.124+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff49010bbd0 con 0x7ff4901094b0 2024-09-21T05:45:30.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.124+0000 7ff4877fe640 1 -- 172.21.15.28:0/223126992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff4800d7560 con 0x7ff4901094b0 2024-09-21T05:45:30.127 INFO:teuthology.orchestra.run.smithi028.stdout:128849018937 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 msgr2=0x7ff47c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 0x7ff47c0b8b80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff478005fd0 tx=0x7ff47800b390 comp rx=0 tx=0).stop 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 msgr2=0x7ff49019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff48002a930 tx=0x7ff480004270 comp rx=0 tx=0).stop 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 shutdown_connections 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff490109e00 0x7ff49019fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff47c0b6670 0x7ff47c0b8b80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4901094b0 0x7ff49019aab0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 --2- 172.21.15.28:0/223126992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff490103b70 0x7ff49019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.126+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 >> 172.21.15.28:0/223126992 conn(0x7ff4900ff5e0 msgr2=0x7ff490101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.127+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 shutdown_connections 2024-09-21T05:45:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:30.127+0000 7ff494da5640 1 -- 172.21.15.28:0/223126992 wait complete. 2024-09-21T05:45:30.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:30 smithi028 ceph-mon[31227]: pgmap v279: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:30.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:30 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/223126992' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T05:45:30.439 INFO:teuthology.orchestra.run.smithi028.stdout:279172874264 2024-09-21T05:45:30.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.11 2024-09-21T05:45:30.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:30 smithi181 ceph-mon[34819]: pgmap v279: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:30.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:30 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/223126992' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T05:45:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:30 smithi129 ceph-mon[34885]: pgmap v279: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 109 B/s, 0 objects/s recovering 2024-09-21T05:45:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:30 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/223126992' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T05:45:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 -- 172.21.15.28:0/2354108514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 msgr2=0x7f7f401087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 --2- 172.21.15.28:0/2354108514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f401087b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009900 tx=0x7f7f3402a410 comp rx=0 tx=0).stop 2024-09-21T05:45:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 -- 172.21.15.28:0/2354108514 shutdown_connections 2024-09-21T05:45:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 --2- 172.21.15.28:0/2354108514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40108cf0 0x7f7f4010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 --2- 172.21.15.28:0/2354108514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f401087b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 --2- 172.21.15.28:0/2354108514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40102a60 0x7f7f40102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 -- 172.21.15.28:0/2354108514 >> 172.21.15.28:0/2354108514 conn(0x7f7f40079730 msgr2=0x7f7f40079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 -- 172.21.15.28:0/2354108514 shutdown_connections 2024-09-21T05:45:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.128+0000 7f7f477b3640 1 -- 172.21.15.28:0/2354108514 wait complete. 2024-09-21T05:45:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 Processor -- start 2024-09-21T05:45:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 -- start start 2024-09-21T05:45:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40102a60 0x7f7f4019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40108cf0 0x7f7f4019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40112960 con 0x7f7f401083a0 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f401127e0 con 0x7f7f40108cf0 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f477b3640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f40112ae0 con 0x7f7f40102a60 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f44d4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f44d4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56270/0 (socket says 172.21.15.28:56270) 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f44d4e640 1 -- 172.21.15.28:0/1437544794 learned_addr learned my addr 172.21.15.28:0/1437544794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f4554f640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40102a60 0x7f7f4019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.129+0000 7f7f45d50640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40108cf0 0x7f7f4019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 -- 172.21.15.28:0/1437544794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40102a60 msgr2=0x7f7f4019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40102a60 0x7f7f4019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 -- 172.21.15.28:0/1437544794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40108cf0 msgr2=0x7f7f4019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40108cf0 0x7f7f4019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 -- 172.21.15.28:0/1437544794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f401a0430 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f44d4e640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f34032fd0 tx=0x7f7f34004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3403f030 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f34004490 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f401a06c0 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34035070 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.130+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f401a6e80 con 0x7f7f401083a0 2024-09-21T05:45:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.131+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f40102e70 con 0x7f7f401083a0 2024-09-21T05:45:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.134+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f7f34042030 con 0x7f7f401083a0 2024-09-21T05:45:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.134+0000 7f7f3e7fc640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 0x7f7f280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.134+0000 7f7f4554f640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 0x7f7f280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.134+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7f34107ef0 con 0x7f7f401083a0 2024-09-21T05:45:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.135+0000 7f7f4554f640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 0x7f7f280b8b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800bbe0 tx=0x7f7f3800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.138+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7f34139710 con 0x7f7f401083a0 2024-09-21T05:45:31.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.332+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7f4010baf0 con 0x7f7f401083a0 2024-09-21T05:45:31.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.332+0000 7f7f3e7fc640 1 -- 172.21.15.28:0/1437544794 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7f340ce520 con 0x7f7f401083a0 2024-09-21T05:45:31.335 INFO:teuthology.orchestra.run.smithi028.stdout:146028888116 2024-09-21T05:45:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.334+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 msgr2=0x7f7f280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.334+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 0x7f7f280b8b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800bbe0 tx=0x7f7f3800b390 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.334+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 msgr2=0x7f7f4019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.334+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f34032fd0 tx=0x7f7f34004270 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 shutdown_connections 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f40108cf0 0x7f7f4019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7f280b65f0 0x7f7f280b8b00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f401083a0 0x7f7f4019aa60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 --2- 172.21.15.28:0/1437544794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f40102a60 0x7f7f4019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 >> 172.21.15.28:0/1437544794 conn(0x7f7f40079730 msgr2=0x7f7f4006b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 shutdown_connections 2024-09-21T05:45:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.335+0000 7f7f477b3640 1 -- 172.21.15.28:0/1437544794 wait complete. 2024-09-21T05:45:32.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/1286744561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f4103b80 msgr2=0x7fc2f4103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/1286744561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f4103b80 0x7fc2f4103f90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009ca0 tx=0x7fc2e402a8e0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/1286744561 shutdown_connections 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/1286744561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 0x7fc2f4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/1286744561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f41094c0 0x7fc2f41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/1286744561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f4103b80 0x7fc2f4103f90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/1286744561 >> 172.21.15.28:0/1286744561 conn(0x7fc2f40ff610 msgr2=0x7fc2f4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/1286744561 shutdown_connections 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:31.999+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/1286744561 wait complete. 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.000+0000 7fc2fc0a5640 1 Processor -- start 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.000+0000 7fc2fc0a5640 1 -- start start 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.000+0000 7fc2fc0a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f4103b80 0x7fc2f419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 0x7fc2f419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4113b90 con 0x7fc2f41094c0 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4113a10 con 0x7fc2f4103b80 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4113d10 con 0x7fc2f4109e10 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56290/0 (socket says 172.21.15.28:56290) 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 -- 172.21.15.28:0/3350854333 learned_addr learned my addr 172.21.15.28:0/3350854333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9e41640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f4103b80 0x7fc2f419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fa642640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 0x7fc2f419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 -- 172.21.15.28:0/3350854333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 msgr2=0x7fc2f419fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 0x7fc2f419fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 -- 172.21.15.28:0/3350854333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f4103b80 msgr2=0x7fc2f419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f4103b80 0x7fc2f419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 -- 172.21.15.28:0/3350854333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f41a0550 con 0x7fc2f41094c0 2024-09-21T05:45:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f9640640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8006fd0 tx=0x7fc2e800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8011830 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e8016070 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f41a0840 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8010e00 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.001+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f41a6fa0 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.002+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f4104010 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.004+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fc2e80119d0 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.005+0000 7fc2f2ffd640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 0x7fc2dc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.005+0000 7fc2f9e41640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 0x7fc2dc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.005+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc2e80e70d0 con 0x7fc2f41094c0 2024-09-21T05:45:32.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.005+0000 7fc2f9e41640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 0x7fc2dc0b8b80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009a60 tx=0x7fc2e40093e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.008+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc2e80ad7a0 con 0x7fc2f41094c0 2024-09-21T05:45:32.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.200+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc2f419b710 con 0x7fc2f41094c0 2024-09-21T05:45:32.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.200+0000 7fc2f2ffd640 1 -- 172.21.15.28:0/3350854333 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc2f419b710 con 0x7fc2f41094c0 2024-09-21T05:45:32.203 INFO:teuthology.orchestra.run.smithi028.stdout:236223201313 2024-09-21T05:45:32.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 msgr2=0x7fc2dc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 0x7fc2dc0b8b80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009a60 tx=0x7fc2e40093e0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 msgr2=0x7fc2f419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8006fd0 tx=0x7fc2e800b8e0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 shutdown_connections 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2f4109e10 0x7fc2f419fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2dc0b6670 0x7fc2dc0b8b80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f41094c0 0x7fc2f419ab80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 --2- 172.21.15.28:0/3350854333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f4103b80 0x7fc2f419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 >> 172.21.15.28:0/3350854333 conn(0x7fc2f40ff610 msgr2=0x7fc2f4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 shutdown_connections 2024-09-21T05:45:32.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.202+0000 7fc2fc0a5640 1 -- 172.21.15.28:0/3350854333 wait complete. 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 -- 172.21.15.28:0/1921692612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8108cf0 msgr2=0x7f52f810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 --2- 172.21.15.28:0/1921692612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8108cf0 0x7f52f810f520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009ca0 tx=0x7f52f002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 -- 172.21.15.28:0/1921692612 shutdown_connections 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 --2- 172.21.15.28:0/1921692612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8108cf0 0x7f52f810f520 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 --2- 172.21.15.28:0/1921692612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f81083a0 0x7f52f81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 --2- 172.21.15.28:0/1921692612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f8102a60 0x7f52f8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 -- 172.21.15.28:0/1921692612 >> 172.21.15.28:0/1921692612 conn(0x7f52f8079730 msgr2=0x7f52f8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 -- 172.21.15.28:0/1921692612 shutdown_connections 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.329+0000 7f52ff753640 1 -- 172.21.15.28:0/1921692612 wait complete. 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.330+0000 7f52ff753640 1 Processor -- start 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.330+0000 7f52ff753640 1 -- start start 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f81083a0 0x7f52f81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f8108cf0 0x7f52f819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8112960 con 0x7f52f8102a60 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81127e0 con 0x7f52f8108cf0 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ff753640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8112ae0 con 0x7f52f81083a0 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56312/0 (socket says 172.21.15.28:56312) 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 -- 172.21.15.28:0/1967084601 learned_addr learned my addr 172.21.15.28:0/1967084601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fdcf0640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f8108cf0 0x7f52f819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fccee640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f81083a0 0x7f52f81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 -- 172.21.15.28:0/1967084601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f81083a0 msgr2=0x7f52f81998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f81083a0 0x7f52f81998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 -- 172.21.15.28:0/1967084601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f8108cf0 msgr2=0x7f52f819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f8108cf0 0x7f52f819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 -- 172.21.15.28:0/1967084601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f819f280 con 0x7f52f8102a60 2024-09-21T05:45:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52fd4ef640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f52f400e9f0 tx=0x7f52f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.331+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f400cd80 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.332+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f819f570 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.332+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52f40054d0 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.332+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4010690 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.332+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81a5cd0 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.333+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f8102e70 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.334+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f52f4004e90 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.335+0000 7f52ea7fc640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 0x7f52e40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.335+0000 7f52fccee640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 0x7f52e40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.335+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f52f40e7ae0 con 0x7f52f8102a60 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.336+0000 7f52fccee640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 0x7f52e40b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f52f8105e80 tx=0x7f52ec005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.339+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f52f40ae0b0 con 0x7f52f8102a60 2024-09-21T05:45:32.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.547+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f52f802e640 con 0x7f52f8102a60 2024-09-21T05:45:32.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:32 smithi028 ceph-mon[31227]: pgmap v280: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:32.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:32 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1437544794' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T05:45:32.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:32 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3350854333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T05:45:32.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.547+0000 7f52ea7fc640 1 -- 172.21.15.28:0/1967084601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f52f40b2490 con 0x7f52f8102a60 2024-09-21T05:45:32.550 INFO:teuthology.orchestra.run.smithi028.stdout:60129542216 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 msgr2=0x7f52e40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 0x7f52e40b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f52f8105e80 tx=0x7f52ec005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 msgr2=0x7f52f8199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f52f400e9f0 tx=0x7f52f400edf0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 shutdown_connections 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f8108cf0 0x7f52f819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52f81083a0 0x7f52f81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52e40b65f0 0x7f52e40b8b00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 --2- 172.21.15.28:0/1967084601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8102a60 0x7f52f8199310 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 >> 172.21.15.28:0/1967084601 conn(0x7f52f8079730 msgr2=0x7f52f81043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 shutdown_connections 2024-09-21T05:45:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.549+0000 7f52ff753640 1 -- 172.21.15.28:0/1967084601 wait complete. 2024-09-21T05:45:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:32 smithi129 ceph-mon[34885]: pgmap v280: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:32 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1437544794' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T05:45:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:32 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3350854333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T05:45:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:32 smithi181 ceph-mon[34819]: pgmap v280: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:32 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1437544794' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T05:45:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:32 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3350854333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T05:45:33.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201311 got 236223201313 for osd.9 2024-09-21T05:45:33.513 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:33.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:33 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/778322378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T05:45:33.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:33 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1967084601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T05:45:33.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:33 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/778322378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T05:45:33.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:33 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1967084601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T05:45:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:33 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/778322378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T05:45:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:33 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1967084601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T05:45:33.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018934 got 128849018937 for osd.4 2024-09-21T05:45:33.963 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:34.329 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888113 got 146028888116 for osd.5 2024-09-21T05:45:34.329 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:34.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:34 smithi028 ceph-mon[31227]: pgmap v281: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-21T05:45:34.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:34 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:45:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:34.601 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:34 smithi181 ceph-mon[34819]: pgmap v281: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-21T05:45:34.601 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:34 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:45:34.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:34 smithi129 ceph-mon[34885]: pgmap v281: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-09-21T05:45:34.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:34 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:45:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 -- 172.21.15.28:0/2427463382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 msgr2=0x7f8f50110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 --2- 172.21.15.28:0/2427463382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50110640 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009ca0 tx=0x7f8f4802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 -- 172.21.15.28:0/2427463382 shutdown_connections 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 --2- 172.21.15.28:0/2427463382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50110640 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 --2- 172.21.15.28:0/2427463382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f501098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 --2- 172.21.15.28:0/2427463382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 -- 172.21.15.28:0/2427463382 >> 172.21.15.28:0/2427463382 conn(0x7f8f500ff610 msgr2=0x7f8f50101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 -- 172.21.15.28:0/2427463382 shutdown_connections 2024-09-21T05:45:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.905+0000 7f8f563fe640 1 -- 172.21.15.28:0/2427463382 wait complete. 2024-09-21T05:45:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.906+0000 7f8f563fe640 1 Processor -- start 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.906+0000 7f8f563fe640 1 -- start start 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f50075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50113ad0 con 0x7f8f50109e10 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50113950 con 0x7f8f50103b80 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f563fe640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50113c50 con 0x7f8f501094c0 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f5499b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f5499b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56332/0 (socket says 172.21.15.28:56332) 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f5499b640 1 -- 172.21.15.28:0/3478212384 learned_addr learned my addr 172.21.15.28:0/3478212384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4f7fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f50075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 -- 172.21.15.28:0/3478212384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 msgr2=0x7f8f50075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f50075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 -- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 msgr2=0x7f8f50076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 -- 172.21.15.28:0/3478212384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f501a8700 con 0x7f8f50103b80 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4f7fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f50075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f5499b640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.907+0000 7f8f4ffff640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50075710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9f0 tx=0x7f8f4000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.908+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4000ccc0 con 0x7f8f50103b80 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.908+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f501a89f0 con 0x7f8f50103b80 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.908+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f400108e0 con 0x7f8f50103b80 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.908+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40018880 con 0x7f8f50103b80 2024-09-21T05:45:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.908+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f501a8f80 con 0x7f8f50103b80 2024-09-21T05:45:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.909+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f50103f90 con 0x7f8f50103b80 2024-09-21T05:45:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.911+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f8f40010430 con 0x7f8f50103b80 2024-09-21T05:45:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.912+0000 7f8f4d7fa640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 0x7f8f380b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.912+0000 7f8f4f7fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 0x7f8f380b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.912+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8f400e7e20 con 0x7f8f50103b80 2024-09-21T05:45:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.913+0000 7f8f4f7fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 0x7f8f380b8b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f50102ef0 tx=0x7f8f44005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:34.914+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8f400b27d0 con 0x7f8f50103b80 2024-09-21T05:45:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.110+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8f5002e640 con 0x7f8f50103b80 2024-09-21T05:45:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.110+0000 7f8f4d7fa640 1 -- 172.21.15.28:0/3478212384 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f400b7710 con 0x7f8f50103b80 2024-09-21T05:45:35.113 INFO:teuthology.orchestra.run.smithi028.stdout:38654705740 2024-09-21T05:45:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 msgr2=0x7f8f380b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 0x7f8f380b8b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f50102ef0 tx=0x7f8f44005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 msgr2=0x7f8f50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50075710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9f0 tx=0x7f8f4000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 shutdown_connections 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f380b65f0 0x7f8f380b8b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f50109e10 0x7f8f50076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f501094c0 0x7f8f50075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 --2- 172.21.15.28:0/3478212384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f50103b80 0x7f8f50075710 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 >> 172.21.15.28:0/3478212384 conn(0x7f8f500ff610 msgr2=0x7f8f50108b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 shutdown_connections 2024-09-21T05:45:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.113+0000 7f8f563fe640 1 -- 172.21.15.28:0/3478212384 wait complete. 2024-09-21T05:45:35.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:35 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3478212384' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T05:45:35.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:35 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542215 got 60129542216 for osd.1 2024-09-21T05:45:35.552 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:35 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3478212384' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T05:45:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:35 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:35 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3478212384' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T05:45:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:35 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 -- 172.21.15.28:0/363933448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108420 msgr2=0x7ff29c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/363933448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108420 0x7ff29c108830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff288009900 tx=0x7ff28802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 -- 172.21.15.28:0/363933448 shutdown_connections 2024-09-21T05:45:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/363933448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108d70 0x7ff29c10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/363933448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108420 0x7ff29c108830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/363933448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 0x7ff29c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 -- 172.21.15.28:0/363933448 >> 172.21.15.28:0/363933448 conn(0x7ff29c079770 msgr2=0x7ff29c079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 -- 172.21.15.28:0/363933448 shutdown_connections 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.859+0000 7ff2a0a90640 1 -- 172.21.15.28:0/363933448 wait complete. 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 Processor -- start 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 -- start start 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 0x7ff29c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108420 0x7ff29c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c113b50 con 0x7ff29c108d70 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1139d0 con 0x7ff29c102ae0 2024-09-21T05:45:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff2a0a90640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c113cd0 con 0x7ff29c108420 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57488/0 (socket says 172.21.15.28:57488) 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 -- 172.21.15.28:0/3077647884 learned_addr learned my addr 172.21.15.28:0/3077647884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29a59c640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 0x7ff29c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff299d9b640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108420 0x7ff29c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 -- 172.21.15.28:0/3077647884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108420 msgr2=0x7ff29c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108420 0x7ff29c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 -- 172.21.15.28:0/3077647884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 msgr2=0x7ff29c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 0x7ff29c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.860+0000 7ff29ad9d640 1 -- 172.21.15.28:0/3077647884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c1a04d0 con 0x7ff29c108d70 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff29ad9d640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eb10 tx=0x7ff29400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294004b20 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff294004cc0 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c1a07c0 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294010670 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.861+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c1a6f20 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.862+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29c103d60 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.864+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7ff294005000 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.865+0000 7ff2937fe640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 0x7ff2800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.865+0000 7ff29a59c640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 0x7ff2800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.865+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff2940e7dc0 con 0x7ff29c108d70 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.866+0000 7ff29a59c640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 0x7ff2800b8b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009bd0 tx=0x7ff28c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:35.868+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff294056710 con 0x7ff29c108d70 2024-09-21T05:45:36.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.059+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff29c102ef0 con 0x7ff29c108d70 2024-09-21T05:45:36.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.059+0000 7ff2937fe640 1 -- 172.21.15.28:0/3077647884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff29c102ef0 con 0x7ff29c108d70 2024-09-21T05:45:36.061 INFO:teuthology.orchestra.run.smithi028.stdout:257698037789 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 msgr2=0x7ff2800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 0x7ff2800b8b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009bd0 tx=0x7ff28c005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 msgr2=0x7ff29c19fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eb10 tx=0x7ff29400c2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 shutdown_connections 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff2800b65f0 0x7ff2800b8b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108d70 0x7ff29c19fda0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff29c108420 0x7ff29c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 --2- 172.21.15.28:0/3077647884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29c102ae0 0x7ff29c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 >> 172.21.15.28:0/3077647884 conn(0x7ff29c079770 msgr2=0x7ff29c106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 shutdown_connections 2024-09-21T05:45:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:36.061+0000 7ff2a0a90640 1 -- 172.21.15.28:0/3077647884 wait complete. 2024-09-21T05:45:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 -- 172.21.15.28:0/2891308709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc10ddf0 msgr2=0x7f5ddc110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 --2- 172.21.15.28:0/2891308709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc110300 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009ca0 tx=0x7f5dd402a8b0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 -- 172.21.15.28:0/2891308709 shutdown_connections 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 --2- 172.21.15.28:0/2891308709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc110300 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 --2- 172.21.15.28:0/2891308709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc105270 0x7f5ddc10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 --2- 172.21.15.28:0/2891308709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc104840 0x7f5ddc104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 -- 172.21.15.28:0/2891308709 >> 172.21.15.28:0/2891308709 conn(0x7f5ddc079730 msgr2=0x7f5ddc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 -- 172.21.15.28:0/2891308709 shutdown_connections 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.227+0000 7f5de308a640 1 -- 172.21.15.28:0/2891308709 wait complete. 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 Processor -- start 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 -- start start 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc105270 0x7f5ddc19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc113a70 con 0x7f5ddc104840 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc1138f0 con 0x7f5ddc105270 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.228+0000 7f5de308a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc113bf0 con 0x7f5ddc10ddf0 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56296/0 (socket says 172.21.15.28:56296) 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 -- 172.21.15.28:0/778322378 learned_addr learned my addr 172.21.15.28:0/778322378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5ddbfff640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc105270 0x7f5ddc19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:36.556 INFO:teuthology.orchestra.run.smithi028.stdout:167503724591 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de1627640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 -- 172.21.15.28:0/778322378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc10ddf0 msgr2=0x7f5ddc19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 -- 172.21.15.28:0/778322378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc105270 msgr2=0x7f5ddc19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc105270 0x7f5ddc19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 -- 172.21.15.28:0/778322378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc1a03b0 con 0x7f5ddc104840 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de0e26640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e9c0 tx=0x7f5dd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd000cd80 con 0x7f5ddc104840 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.229+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc1a06a0 con 0x7f5ddc104840 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.230+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc1a6e00 con 0x7f5ddc104840 2024-09-21T05:45:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.230+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dd00035f0 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.230+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0010690 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.231+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc00052c0 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.232+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f5dd0004000 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.233+0000 7f5dd9ffb640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 0x7f5dcc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.233+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5dd00e7950 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.234+0000 7f5ddbfff640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 0x7f5dcc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.234+0000 7f5ddbfff640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 0x7f5dcc0b8b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc10fae0 tx=0x7f5dc8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.238+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5dd00adf20 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.453+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5dc0005880 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.454+0000 7f5dd9ffb640 1 -- 172.21.15.28:0/778322378 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5dd00b2300 con 0x7f5ddc104840 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 msgr2=0x7f5dcc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 0x7f5dcc0b8b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc10fae0 tx=0x7f5dc8005f90 comp rx=0 tx=0).stop 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 msgr2=0x7f5ddc19a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e9c0 tx=0x7f5dd000ee80 comp rx=0 tx=0).stop 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 shutdown_connections 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ddc10ddf0 0x7f5ddc19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ddc105270 0x7f5ddc19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5dcc0b6670 0x7f5dcc0b8b80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 --2- 172.21.15.28:0/778322378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ddc104840 0x7f5ddc19a440 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 >> 172.21.15.28:0/778322378 conn(0x7f5ddc079730 msgr2=0x7f5ddc0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 shutdown_connections 2024-09-21T05:45:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:32.456+0000 7f5de308a640 1 -- 172.21.15.28:0/778322378 wait complete. 2024-09-21T05:45:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:36 smithi129 ceph-mon[34885]: pgmap v282: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:45:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:36 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3077647884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T05:45:36.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:36 smithi181 ceph-mon[34819]: pgmap v282: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:45:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:36 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3077647884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T05:45:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:36 smithi028 ceph-mon[31227]: pgmap v282: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:45:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3077647884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T05:45:37.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724589 got 167503724591 for osd.6 2024-09-21T05:45:37.121 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 -- 172.21.15.28:0/1097011871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8108cf0 msgr2=0x7ff1b810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/1097011871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8108cf0 0x7ff1b810f520 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009ca0 tx=0x7ff1b002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 -- 172.21.15.28:0/1097011871 shutdown_connections 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/1097011871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8108cf0 0x7ff1b810f520 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/1097011871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b81083a0 0x7ff1b81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/1097011871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b8102a60 0x7ff1b8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 -- 172.21.15.28:0/1097011871 >> 172.21.15.28:0/1097011871 conn(0x7ff1b8079730 msgr2=0x7ff1b8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 -- 172.21.15.28:0/1097011871 shutdown_connections 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.781+0000 7ff1bcaee640 1 -- 172.21.15.28:0/1097011871 wait complete. 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 Processor -- start 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 -- start start 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b81083a0 0x7ff1b819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b8108cf0 0x7ff1b81a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8112960 con 0x7ff1b8102a60 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81127e0 con 0x7ff1b8108cf0 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1bcaee640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8112ae0 con 0x7ff1b81083a0 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57538/0 (socket says 172.21.15.28:57538) 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.782+0000 7ff1b659c640 1 -- 172.21.15.28:0/4186506436 learned_addr learned my addr 172.21.15.28:0/4186506436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b6d9d640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b8108cf0 0x7ff1b81a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b5d9b640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b81083a0 0x7ff1b819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 -- 172.21.15.28:0/4186506436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b81083a0 msgr2=0x7ff1b819cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b81083a0 0x7ff1b819cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 -- 172.21.15.28:0/4186506436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b8108cf0 msgr2=0x7ff1b81a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b8108cf0 0x7ff1b81a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 -- 172.21.15.28:0/4186506436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b81a2570 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1b659c640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800d770 tx=0x7ff1a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8017070 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a80054d0 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b81a2860 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8004c30 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.783+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b81a8fc0 con 0x7ff1b8102a60 2024-09-21T05:45:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.785+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b8102e70 con 0x7ff1b8102a60 2024-09-21T05:45:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.787+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7ff1a8004dd0 con 0x7ff1b8102a60 2024-09-21T05:45:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.788+0000 7ff1a77fe640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 0x7ff1a00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.789+0000 7ff1b5d9b640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 0x7ff1a00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.789+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff1a80e7450 con 0x7ff1b8102a60 2024-09-21T05:45:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.789+0000 7ff1b5d9b640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 0x7ff1a00b8b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1b806a300 tx=0x7ff1ac00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.791+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff1a80b1e00 con 0x7ff1b8102a60 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 -- 172.21.15.28:0/3216778402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 msgr2=0x7f8244104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 --2- 172.21.15.28:0/3216778402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f8244104c50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8240009900 tx=0x7f824002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 -- 172.21.15.28:0/3216778402 shutdown_connections 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 --2- 172.21.15.28:0/3216778402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 0x7f8244110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 --2- 172.21.15.28:0/3216778402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 --2- 172.21.15.28:0/3216778402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f8244104c50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 -- 172.21.15.28:0/3216778402 >> 172.21.15.28:0/3216778402 conn(0x7f8244079730 msgr2=0x7f8244079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 -- 172.21.15.28:0/3216778402 shutdown_connections 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.704+0000 7f824b7e5640 1 -- 172.21.15.28:0/3216778402 wait complete. 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.705+0000 7f824b7e5640 1 Processor -- start 2024-09-21T05:45:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.705+0000 7f824b7e5640 1 -- start start 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 0x7f824419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8244113a70 con 0x7f8244104840 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82441138f0 con 0x7f8244105270 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f824b7e5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8244113bf0 con 0x7f824410ddf0 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57512/0 (socket says 172.21.15.28:57512) 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 -- 172.21.15.28:0/586262927 learned_addr learned my addr 172.21.15.28:0/586262927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249d82640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 0x7f824419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8248d80640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 -- 172.21.15.28:0/586262927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 msgr2=0x7f824419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 0x7f824419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 -- 172.21.15.28:0/586262927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 msgr2=0x7f824419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 -- 172.21.15.28:0/586262927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82441a0430 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8248d80640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.706+0000 7f8249581640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8240006010 tx=0x7f8240004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.707+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824003f030 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.707+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82441a06c0 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.707+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8240003aa0 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.707+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240035070 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.707+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82441a6e80 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.708+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824406b500 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.710+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f8240042030 con 0x7f8244104840 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.710+0000 7f823a7fc640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 0x7f82300b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.710+0000 7f8248d80640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 0x7f82300b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:37.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.710+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8240107fa0 con 0x7f8244104840 2024-09-21T05:45:37.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.711+0000 7f8248d80640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 0x7f82300b8b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8234005d00 tx=0x7f8234005ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:37.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.714+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f82400ce570 con 0x7f8244104840 2024-09-21T05:45:37.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.910+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f82441a7080 con 0x7f8244104840 2024-09-21T05:45:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.910+0000 7f823a7fc640 1 -- 172.21.15.28:0/586262927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82400d2950 con 0x7f8244104840 2024-09-21T05:45:37.912 INFO:teuthology.orchestra.run.smithi028.stdout:77309411397 2024-09-21T05:45:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 msgr2=0x7f82300b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 0x7f82300b8b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8234005d00 tx=0x7f8234005ed0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 msgr2=0x7f824419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8240006010 tx=0x7f8240004060 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 shutdown_connections 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f824410ddf0 0x7f824419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8244105270 0x7f824419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82300b65f0 0x7f82300b8b00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 --2- 172.21.15.28:0/586262927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8244104840 0x7f824419a4c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.913+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 >> 172.21.15.28:0/586262927 conn(0x7f8244079730 msgr2=0x7f8244107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.914+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 shutdown_connections 2024-09-21T05:45:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.914+0000 7f824b7e5640 1 -- 172.21.15.28:0/586262927 wait complete. 2024-09-21T05:45:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.983+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff1b81a91f0 con 0x7ff1b8102a60 2024-09-21T05:45:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.984+0000 7ff1a77fe640 1 -- 172.21.15.28:0/4186506436 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff1a80b6d40 con 0x7ff1b8102a60 2024-09-21T05:45:37.986 INFO:teuthology.orchestra.run.smithi028.stdout:103079215167 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 msgr2=0x7ff1a00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 0x7ff1a00b8b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1b806a300 tx=0x7ff1ac00b390 comp rx=0 tx=0).stop 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 msgr2=0x7ff1b819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800d770 tx=0x7ff1a800dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 shutdown_connections 2024-09-21T05:45:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b8108cf0 0x7ff1b81a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1b81083a0 0x7ff1b819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1a00b6670 0x7ff1a00b8b80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 --2- 172.21.15.28:0/4186506436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1b8102a60 0x7ff1b819c600 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 >> 172.21.15.28:0/4186506436 conn(0x7ff1b8079730 msgr2=0x7ff1b806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 shutdown_connections 2024-09-21T05:45:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:37.986+0000 7ff1bcaee640 1 -- 172.21.15.28:0/4186506436 wait complete. 2024-09-21T05:45:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:38 smithi129 ceph-mon[34885]: pgmap v283: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/586262927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T05:45:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:38 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4186506436' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T05:45:38.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:38 smithi028 ceph-mon[31227]: pgmap v283: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:38.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/586262927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T05:45:38.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:38 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4186506436' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T05:45:38.691 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411397 for osd.2 2024-09-21T05:45:38.691 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:38 smithi181 ceph-mon[34819]: pgmap v283: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/586262927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T05:45:38.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:38 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4186506436' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.929+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3051698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f101083d0 msgr2=0x7f7f101087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.929+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3051698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f101083d0 0x7f7f101087e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009900 tx=0x7f7f0402a410 comp rx=0 tx=0).stop 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3051698004 shutdown_connections 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3051698004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f10108d20 0x7f7f1010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3051698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f101083d0 0x7f7f101087e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3051698004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f10102ae0 0x7f7f10102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3051698004 >> 172.21.15.28:0/3051698004 conn(0x7f7f100797b0 msgr2=0x7f7f10079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3051698004 shutdown_connections 2024-09-21T05:45:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3051698004 wait complete. 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.930+0000 7f7f14ab1640 1 Processor -- start 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 -- start start 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f101083d0 0x7f7f100ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f10108d20 0x7f7f101000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101129e0 con 0x7f7f10102ae0 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10112860 con 0x7f7f101083d0 2024-09-21T05:45:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f14ab1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10112b60 con 0x7f7f10108d20 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57562/0 (socket says 172.21.15.28:57562) 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 -- 172.21.15.28:0/3241663557 learned_addr learned my addr 172.21.15.28:0/3241663557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0dd9b640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f101083d0 0x7f7f100ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 -- 172.21.15.28:0/3241663557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f10108d20 msgr2=0x7f7f101000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f10108d20 0x7f7f101000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 -- 172.21.15.28:0/3241663557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f101083d0 msgr2=0x7f7f100ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f101083d0 0x7f7f100ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.931+0000 7f7f0e59c640 1 -- 172.21.15.28:0/3241663557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f101a8690 con 0x7f7f10102ae0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7f0e59c640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000cce0 tx=0x7f7f0000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0000f650 con 0x7f7f10102ae0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0000fca0 con 0x7f7f10102ae0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f101a8980 con 0x7f7f10102ae0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00018b40 con 0x7f7f10102ae0 2024-09-21T05:45:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.932+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f101a8f10 con 0x7f7f10102ae0 2024-09-21T05:45:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.933+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f10102ef0 con 0x7f7f10102ae0 2024-09-21T05:45:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.935+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f7f00018d00 con 0x7f7f10102ae0 2024-09-21T05:45:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.936+0000 7f7eff7fe640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 0x7f7ef80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.936+0000 7f7f0dd9b640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 0x7f7ef80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.936+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7f000e7e00 con 0x7f7f10102ae0 2024-09-21T05:45:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.937+0000 7f7f0dd9b640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 0x7f7ef80b8b80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006010 tx=0x7f7f04000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:38.939+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7f00014030 con 0x7f7f10102ae0 2024-09-21T05:45:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.132+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f7f10100970 con 0x7f7f10102ae0 2024-09-21T05:45:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.132+0000 7f7eff7fe640 1 -- 172.21.15.28:0/3241663557 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7f000ae540 con 0x7f7f10102ae0 2024-09-21T05:45:39.134 INFO:teuthology.orchestra.run.smithi028.stdout:188978561068 2024-09-21T05:45:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 msgr2=0x7f7ef80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 0x7f7ef80b8b80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006010 tx=0x7f7f04000bc0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 msgr2=0x7f7f100ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000cce0 tx=0x7f7f0000a620 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 shutdown_connections 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f10108d20 0x7f7f101000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f101083d0 0x7f7f100ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7ef80b6670 0x7f7ef80b8b80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 --2- 172.21.15.28:0/3241663557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f10102ae0 0x7f7f100ff670 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.134+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 >> 172.21.15.28:0/3241663557 conn(0x7f7f100797b0 msgr2=0x7f7f10101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.135+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 shutdown_connections 2024-09-21T05:45:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.135+0000 7f7f14ab1640 1 -- 172.21.15.28:0/3241663557 wait complete. 2024-09-21T05:45:39.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037786 got 257698037789 for osd.10 2024-09-21T05:45:39.370 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 -- 172.21.15.28:0/1586297545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 msgr2=0x7fda9c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 --2- 172.21.15.28:0/1586297545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c10f520 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fda94009ca0 tx=0x7fda9402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 -- 172.21.15.28:0/1586297545 shutdown_connections 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 --2- 172.21.15.28:0/1586297545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c10f520 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 --2- 172.21.15.28:0/1586297545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c1083a0 0x7fda9c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 --2- 172.21.15.28:0/1586297545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c102a60 0x7fda9c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 -- 172.21.15.28:0/1586297545 >> 172.21.15.28:0/1586297545 conn(0x7fda9c079730 msgr2=0x7fda9c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 -- 172.21.15.28:0/1586297545 shutdown_connections 2024-09-21T05:45:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.548+0000 7fdaa2517640 1 -- 172.21.15.28:0/1586297545 wait complete. 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 Processor -- start 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 -- start start 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c102a60 0x7fda9c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c1083a0 0x7fda9c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113a80 con 0x7fda9c108cf0 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113900 con 0x7fda9c1083a0 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa2517640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113c00 con 0x7fda9c102a60 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa0ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa0ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57580/0 (socket says 172.21.15.28:57580) 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.549+0000 7fdaa0ab4640 1 -- 172.21.15.28:0/542959863 learned_addr learned my addr 172.21.15.28:0/542959863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fda9bfff640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c102a60 0x7fda9c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fda9b7fe640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c1083a0 0x7fda9c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 -- 172.21.15.28:0/542959863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c102a60 msgr2=0x7fda9c19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c102a60 0x7fda9c19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 -- 172.21.15.28:0/542959863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c1083a0 msgr2=0x7fda9c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c1083a0 0x7fda9c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 -- 172.21.15.28:0/542959863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c1a0280 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa0ab4640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fda94005cc0 tx=0x7fda9402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94003bf0 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda94003d90 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c1a0510 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9402fb20 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.550+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a0a50 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.551+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c102e70 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.553+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fda94004580 con 0x7fda9c108cf0 2024-09-21T05:45:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.554+0000 7fda997fa640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 0x7fda840b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.554+0000 7fda9bfff640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 0x7fda840b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.554+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fda94108a90 con 0x7fda9c108cf0 2024-09-21T05:45:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.554+0000 7fda9bfff640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 0x7fda840b8b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00abe0 tx=0x7fda8c00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.557+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fda940cf0c0 con 0x7fda9c108cf0 2024-09-21T05:45:39.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:39 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3241663557' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T05:45:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.748+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fda9c19b500 con 0x7fda9c108cf0 2024-09-21T05:45:39.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:39 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3241663557' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T05:45:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.749+0000 7fda997fa640 1 -- 172.21.15.28:0/542959863 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fda9c19b500 con 0x7fda9c108cf0 2024-09-21T05:45:39.751 INFO:teuthology.orchestra.run.smithi028.stdout:214748364839 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 msgr2=0x7fda840b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 0x7fda840b8b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00abe0 tx=0x7fda8c00a2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 msgr2=0x7fda9c19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fda94005cc0 tx=0x7fda9402f380 comp rx=0 tx=0).stop 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 shutdown_connections 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda840b6620 0x7fda840b8b30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108cf0 0x7fda9c19fc10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda9c1083a0 0x7fda9c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 --2- 172.21.15.28:0/542959863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda9c102a60 0x7fda9c19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 >> 172.21.15.28:0/542959863 conn(0x7fda9c079730 msgr2=0x7fda9c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 shutdown_connections 2024-09-21T05:45:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:39.751+0000 7fdaa2517640 1 -- 172.21.15.28:0/542959863 wait complete. 2024-09-21T05:45:39.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:39 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3241663557' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T05:45:40.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215163 got 103079215167 for osd.3 2024-09-21T05:45:40.006 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:40.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705737 got 38654705740 for osd.0 2024-09-21T05:45:40.436 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:40 smithi129 ceph-mon[34885]: pgmap v284: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:40 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/542959863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T05:45:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:40 smithi028 ceph-mon[31227]: pgmap v284: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:40 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/542959863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T05:45:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:40 smithi181 ceph-mon[34819]: pgmap v284: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:40.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:40 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/542959863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T05:45:41.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561065 got 188978561068 for osd.7 2024-09-21T05:45:41.036 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:41.278 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:41.720 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364836 got 214748364839 for osd.8 2024-09-21T05:45:41.720 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2588195500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083b0 msgr2=0x7fd5e41087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2588195500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083b0 0x7fd5e41087c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009900 tx=0x7fd5d802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2588195500 shutdown_connections 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2588195500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4108d00 0x7fd5e410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2588195500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083b0 0x7fd5e41087c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2588195500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e4102a70 0x7fd5e4102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2588195500 >> 172.21.15.28:0/2588195500 conn(0x7fd5e4079730 msgr2=0x7fd5e4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2588195500 shutdown_connections 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.152+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2588195500 wait complete. 2024-09-21T05:45:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 Processor -- start 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 -- start start 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4102a70 0x7fd5e410ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e41083b0 0x7fd5e4110500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4113a90 con 0x7fd5e4108d00 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4113910 con 0x7fd5e4102a70 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5ea3b5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4113c10 con 0x7fd5e41083b0 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.153+0000 7fd5e8952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57596/0 (socket says 172.21.15.28:57596) 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 -- 172.21.15.28:0/2464459094 learned_addr learned my addr 172.21.15.28:0/2464459094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e3fff640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4102a70 0x7fd5e410ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e37fe640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e41083b0 0x7fd5e4110500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 -- 172.21.15.28:0/2464459094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e41083b0 msgr2=0x7fd5e4110500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e41083b0 0x7fd5e4110500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 -- 172.21.15.28:0/2464459094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4102a70 msgr2=0x7fd5e410ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4102a70 0x7fd5e410ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 -- 172.21.15.28:0/2464459094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e41a8760 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e8952640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00eb10 tx=0x7fd5dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc004b20 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5dc004cc0 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e41a8a50 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc010670 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.154+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e41a8f10 con 0x7fd5e4108d00 2024-09-21T05:45:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.155+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e4103cf0 con 0x7fd5e4108d00 2024-09-21T05:45:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.157+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fd5dc005000 con 0x7fd5e4108d00 2024-09-21T05:45:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.158+0000 7fd5e17fa640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 0x7fd5cc0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.158+0000 7fd5e3fff640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 0x7fd5cc0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.158+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd5dc0e7e20 con 0x7fd5e4108d00 2024-09-21T05:45:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.158+0000 7fd5e3fff640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 0x7fd5cc0b8b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4111a10 tx=0x7fd5d400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.161+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd5dc056710 con 0x7fd5e4108d00 2024-09-21T05:45:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.364+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd5e4102e80 con 0x7fd5e4108d00 2024-09-21T05:45:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.365+0000 7fd5e17fa640 1 -- 172.21.15.28:0/2464459094 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd5e4102e80 con 0x7fd5e4108d00 2024-09-21T05:45:42.367 INFO:teuthology.orchestra.run.smithi028.stdout:279172874266 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 msgr2=0x7fd5cc0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 0x7fd5cc0b8b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4111a10 tx=0x7fd5d400b390 comp rx=0 tx=0).stop 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 msgr2=0x7fd5e4110a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00eb10 tx=0x7fd5dc00c2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 shutdown_connections 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5cc0b65f0 0x7fd5cc0b8b00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e4108d00 0x7fd5e4110a40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5e41083b0 0x7fd5e4110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 --2- 172.21.15.28:0/2464459094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5e4102a70 0x7fd5e410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 >> 172.21.15.28:0/2464459094 conn(0x7fd5e4079730 msgr2=0x7fd5e4105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 shutdown_connections 2024-09-21T05:45:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:42.367+0000 7fd5ea3b5640 1 -- 172.21.15.28:0/2464459094 wait complete. 2024-09-21T05:45:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:42 smithi129 ceph-mon[34885]: pgmap v285: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:42.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:42 smithi028 ceph-mon[31227]: pgmap v285: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:42 smithi181 ceph-mon[34819]: pgmap v285: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:43.211 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874264 got 279172874266 for osd.11 2024-09-21T05:45:43.211 DEBUG:teuthology.parallel:result is None 2024-09-21T05:45:43.211 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T05:45:43.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T05:45:43.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:43 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2464459094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T05:45:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:43.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:43 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2464459094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T05:45:43.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:43 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2464459094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 -- 172.21.15.28:0/4018744156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8102a80 msgr2=0x7f01e8102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 --2- 172.21.15.28:0/4018744156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8102a80 0x7f01e8102e90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009ca0 tx=0x7f01d802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 -- 172.21.15.28:0/4018744156 shutdown_connections 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 --2- 172.21.15.28:0/4018744156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e8108d10 0x7f01e810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 --2- 172.21.15.28:0/4018744156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e81083c0 0x7f01e81087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 --2- 172.21.15.28:0/4018744156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8102a80 0x7f01e8102e90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 -- 172.21.15.28:0/4018744156 >> 172.21.15.28:0/4018744156 conn(0x7f01e8079740 msgr2=0x7f01e8079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 -- 172.21.15.28:0/4018744156 shutdown_connections 2024-09-21T05:45:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.009+0000 7f01ed897640 1 -- 172.21.15.28:0/4018744156 wait complete. 2024-09-21T05:45:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 Processor -- start 2024-09-21T05:45:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 -- start start 2024-09-21T05:45:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e8102a80 0x7f01e819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e81083c0 0x7f01e819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01e8113a90 con 0x7f01e8108d10 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01e8113910 con 0x7f01e8102a80 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01ed897640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f01e8113c10 con 0x7f01e81083c0 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57616/0 (socket says 172.21.15.28:57616) 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01e77fe640 1 -- 172.21.15.28:0/3702480263 learned_addr learned my addr 172.21.15.28:0/3702480263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.010+0000 7f01e67fc640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e81083c0 0x7f01e819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e6ffd640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e8102a80 0x7f01e819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 -- 172.21.15.28:0/3702480263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e81083c0 msgr2=0x7f01e819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e81083c0 0x7f01e819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 -- 172.21.15.28:0/3702480263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e8102a80 msgr2=0x7f01e819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e8102a80 0x7f01e819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 -- 172.21.15.28:0/3702480263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e81a0370 con 0x7f01e8108d10 2024-09-21T05:45:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01e77fe640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f01e000d750 tx=0x7f01e000db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0017070 con 0x7f01e8108d10 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01e00097c0 con 0x7f01e8108d10 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e81a0660 con 0x7f01e8108d10 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0004ba0 con 0x7f01e8108d10 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.011+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e81a6e80 con 0x7f01e8108d10 2024-09-21T05:45:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.012+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e8103d00 con 0x7f01e8108d10 2024-09-21T05:45:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.014+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f01e0009c90 con 0x7f01e8108d10 2024-09-21T05:45:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.015+0000 7f01ec895640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 0x7f01d00b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.015+0000 7f01e6ffd640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 0x7f01d00b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.015+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f01e00e7420 con 0x7f01e8108d10 2024-09-21T05:45:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.015+0000 7f01e6ffd640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 0x7f01d00b8b30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000f80 tx=0x7f01d8009360 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.019+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f01e00ad9f0 con 0x7f01e8108d10 2024-09-21T05:45:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.285+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01e8063800 con 0x7f01d00b6620 2024-09-21T05:45:44.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.289+0000 7f01ec895640 1 -- 172.21.15.28:0/3702480263 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258757 (secure 0 0 0) 0x7f01e8063800 con 0x7f01d00b6620 2024-09-21T05:45:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:44.295 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 msgr2=0x7f01d00b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 0x7f01d00b8b30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000f80 tx=0x7f01d8009360 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 msgr2=0x7f01e819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f01e000d750 tx=0x7f01e000db50 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 shutdown_connections 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f01d00b6620 0x7f01d00b8b30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e8108d10 0x7f01e819fd00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01e81083c0 0x7f01e819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 --2- 172.21.15.28:0/3702480263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01e8102a80 0x7f01e819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 >> 172.21.15.28:0/3702480263 conn(0x7f01e8079740 msgr2=0x7f01e81014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 shutdown_connections 2024-09-21T05:45:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:44.293+0000 7f01ed897640 1 -- 172.21.15.28:0/3702480263 wait complete. 2024-09-21T05:45:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:44 smithi028 ceph-mon[31227]: pgmap v286: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:44 smithi129 ceph-mon[34885]: pgmap v286: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:44 smithi181 ceph-mon[34819]: pgmap v286: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:44.943 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-21T05:45:42.726965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662172,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":604,"num_read_kb":414,"num_write":366,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2953156,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385928,"kb_used_data":11640,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555896,"statfs":{"total":1151940427776,"available":1151545237504,"internally_reserved":0,"allocated":11919360,"data_stored":6758892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003460"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256259+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:45:14.256259+0000","last_peered":"2024-09-21T05:45:14.256259+0000","last_clean":"2024-09-21T05:45:14.256259+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:45:14.256259+0000","last_undegraded":"2024-09-21T05:45:14.256259+0000","last_fullsized":"2024-09-21T05:45:14.256259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260266+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:45:14.260266+0000","last_peered":"2024-09-21T05:45:14.260266+0000","last_clean":"2024-09-21T05:45:14.260266+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:45:14.260266+0000","last_undegraded":"2024-09-21T05:45:14.260266+0000","last_fullsized":"2024-09-21T05:45:14.260266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260390+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:45:14.260390+0000","last_peered":"2024-09-21T05:45:14.260390+0000","last_clean":"2024-09-21T05:45:14.260390+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:45:14.260390+0000","last_undegraded":"2024-09-21T05:45:14.260390+0000","last_fullsized":"2024-09-21T05:45:14.260390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253684+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:45:14.253684+0000","last_peered":"2024-09-21T05:45:14.253684+0000","last_clean":"2024-09-21T05:45:14.253684+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:45:14.253684+0000","last_undegraded":"2024-09-21T05:45:14.253684+0000","last_fullsized":"2024-09-21T05:45:14.253684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258728+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:45:14.258728+0000","last_peered":"2024-09-21T05:45:14.258728+0000","last_clean":"2024-09-21T05:45:14.258728+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:45:14.258728+0000","last_undegraded":"2024-09-21T05:45:14.258728+0000","last_fullsized":"2024-09-21T05:45:14.258728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259120+0000","last_change":"2024-09-21T05:45:14.258964+0000","last_active":"2024-09-21T05:45:14.259120+0000","last_peered":"2024-09-21T05:45:14.259120+0000","last_clean":"2024-09-21T05:45:14.259120+0000","last_became_active":"2024-09-21T05:45:14.257864+0000","last_became_peered":"2024-09-21T05:45:14.257864+0000","last_unstale":"2024-09-21T05:45:14.259120+0000","last_undegraded":"2024-09-21T05:45:14.259120+0000","last_fullsized":"2024-09-21T05:45:14.259120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.693452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256026+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:45:14.256026+0000","last_peered":"2024-09-21T05:45:14.256026+0000","last_clean":"2024-09-21T05:45:14.256026+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:45:14.256026+0000","last_undegraded":"2024-09-21T05:45:14.256026+0000","last_fullsized":"2024-09-21T05:45:14.256026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321491+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:45:14.321491+0000","last_peered":"2024-09-21T05:45:14.321491+0000","last_clean":"2024-09-21T05:45:14.321491+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:45:14.321491+0000","last_undegraded":"2024-09-21T05:45:14.321491+0000","last_fullsized":"2024-09-21T05:45:14.321491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262607+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:45:14.262607+0000","last_peered":"2024-09-21T05:45:14.262607+0000","last_clean":"2024-09-21T05:45:14.262607+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:45:14.262607+0000","last_undegraded":"2024-09-21T05:45:14.262607+0000","last_fullsized":"2024-09-21T05:45:14.262607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255408+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:45:14.255408+0000","last_peered":"2024-09-21T05:45:14.255408+0000","last_clean":"2024-09-21T05:45:14.255408+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:45:14.255408+0000","last_undegraded":"2024-09-21T05:45:14.255408+0000","last_fullsized":"2024-09-21T05:45:14.255408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256306+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:45:14.256306+0000","last_peered":"2024-09-21T05:45:14.256306+0000","last_clean":"2024-09-21T05:45:14.256306+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:45:14.256306+0000","last_undegraded":"2024-09-21T05:45:14.256306+0000","last_fullsized":"2024-09-21T05:45:14.256306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253789+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:45:14.253789+0000","last_peered":"2024-09-21T05:45:14.253789+0000","last_clean":"2024-09-21T05:45:14.253789+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:45:14.253789+0000","last_undegraded":"2024-09-21T05:45:14.253789+0000","last_fullsized":"2024-09-21T05:45:14.253789+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252491+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:45:14.252491+0000","last_peered":"2024-09-21T05:45:14.252491+0000","last_clean":"2024-09-21T05:45:14.252491+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:45:14.252491+0000","last_undegraded":"2024-09-21T05:45:14.252491+0000","last_fullsized":"2024-09-21T05:45:14.252491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260165+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:45:14.260165+0000","last_peered":"2024-09-21T05:45:14.260165+0000","last_clean":"2024-09-21T05:45:14.260165+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:45:14.260165+0000","last_undegraded":"2024-09-21T05:45:14.260165+0000","last_fullsized":"2024-09-21T05:45:14.260165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260121+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:45:14.260121+0000","last_peered":"2024-09-21T05:45:14.260121+0000","last_clean":"2024-09-21T05:45:14.260121+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:45:14.260121+0000","last_undegraded":"2024-09-21T05:45:14.260121+0000","last_fullsized":"2024-09-21T05:45:14.260121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262111+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:45:14.262111+0000","last_peered":"2024-09-21T05:45:14.262111+0000","last_clean":"2024-09-21T05:45:14.262111+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:45:14.262111+0000","last_undegraded":"2024-09-21T05:45:14.262111+0000","last_fullsized":"2024-09-21T05:45:14.262111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256963+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:45:14.256963+0000","last_peered":"2024-09-21T05:45:14.256963+0000","last_clean":"2024-09-21T05:45:14.256963+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:45:14.256963+0000","last_undegraded":"2024-09-21T05:45:14.256963+0000","last_fullsized":"2024-09-21T05:45:14.256963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255456+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:45:14.255456+0000","last_peered":"2024-09-21T05:45:14.255456+0000","last_clean":"2024-09-21T05:45:14.255456+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:45:14.255456+0000","last_undegraded":"2024-09-21T05:45:14.255456+0000","last_fullsized":"2024-09-21T05:45:14.255456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256579+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:45:14.256579+0000","last_peered":"2024-09-21T05:45:14.256579+0000","last_clean":"2024-09-21T05:45:14.256579+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:45:14.256579+0000","last_undegraded":"2024-09-21T05:45:14.256579+0000","last_fullsized":"2024-09-21T05:45:14.256579+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263278+0000","last_change":"2024-09-21T05:45:14.263089+0000","last_active":"2024-09-21T05:45:14.263278+0000","last_peered":"2024-09-21T05:45:14.263278+0000","last_clean":"2024-09-21T05:45:14.263278+0000","last_became_active":"2024-09-21T05:45:14.262626+0000","last_became_peered":"2024-09-21T05:45:14.262626+0000","last_unstale":"2024-09-21T05:45:14.263278+0000","last_undegraded":"2024-09-21T05:45:14.263278+0000","last_fullsized":"2024-09-21T05:45:14.263278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:34.442511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252627+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:45:14.252627+0000","last_peered":"2024-09-21T05:45:14.252627+0000","last_clean":"2024-09-21T05:45:14.252627+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:45:14.252627+0000","last_undegraded":"2024-09-21T05:45:14.252627+0000","last_fullsized":"2024-09-21T05:45:14.252627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262734+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:45:14.262734+0000","last_peered":"2024-09-21T05:45:14.262734+0000","last_clean":"2024-09-21T05:45:14.262734+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:45:14.262734+0000","last_undegraded":"2024-09-21T05:45:14.262734+0000","last_fullsized":"2024-09-21T05:45:14.262734+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321494+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:45:14.321494+0000","last_peered":"2024-09-21T05:45:14.321494+0000","last_clean":"2024-09-21T05:45:14.321494+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:45:14.321494+0000","last_undegraded":"2024-09-21T05:45:14.321494+0000","last_fullsized":"2024-09-21T05:45:14.321494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252852+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:45:14.252852+0000","last_peered":"2024-09-21T05:45:14.252852+0000","last_clean":"2024-09-21T05:45:14.252852+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:45:14.252852+0000","last_undegraded":"2024-09-21T05:45:14.252852+0000","last_fullsized":"2024-09-21T05:45:14.252852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321687+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:45:14.321687+0000","last_peered":"2024-09-21T05:45:14.321687+0000","last_clean":"2024-09-21T05:45:14.321687+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:45:14.321687+0000","last_undegraded":"2024-09-21T05:45:14.321687+0000","last_fullsized":"2024-09-21T05:45:14.321687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257077+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:45:14.257077+0000","last_peered":"2024-09-21T05:45:14.257077+0000","last_clean":"2024-09-21T05:45:14.257077+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:45:14.257077+0000","last_undegraded":"2024-09-21T05:45:14.257077+0000","last_fullsized":"2024-09-21T05:45:14.257077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256239+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:45:14.256239+0000","last_peered":"2024-09-21T05:45:14.256239+0000","last_clean":"2024-09-21T05:45:14.256239+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:45:14.256239+0000","last_undegraded":"2024-09-21T05:45:14.256239+0000","last_fullsized":"2024-09-21T05:45:14.256239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252494+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:45:14.252494+0000","last_peered":"2024-09-21T05:45:14.252494+0000","last_clean":"2024-09-21T05:45:14.252494+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:45:14.252494+0000","last_undegraded":"2024-09-21T05:45:14.252494+0000","last_fullsized":"2024-09-21T05:45:14.252494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260469+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:45:14.260469+0000","last_peered":"2024-09-21T05:45:14.260469+0000","last_clean":"2024-09-21T05:45:14.260469+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:45:14.260469+0000","last_undegraded":"2024-09-21T05:45:14.260469+0000","last_fullsized":"2024-09-21T05:45:14.260469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256487+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:45:14.256487+0000","last_peered":"2024-09-21T05:45:14.256487+0000","last_clean":"2024-09-21T05:45:14.256487+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:45:14.256487+0000","last_undegraded":"2024-09-21T05:45:14.256487+0000","last_fullsized":"2024-09-21T05:45:14.256487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256369+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:45:14.256369+0000","last_peered":"2024-09-21T05:45:14.256369+0000","last_clean":"2024-09-21T05:45:14.256369+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:45:14.256369+0000","last_undegraded":"2024-09-21T05:45:14.256369+0000","last_fullsized":"2024-09-21T05:45:14.256369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256613+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:45:14.256613+0000","last_peered":"2024-09-21T05:45:14.256613+0000","last_clean":"2024-09-21T05:45:14.256613+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:45:14.256613+0000","last_undegraded":"2024-09-21T05:45:14.256613+0000","last_fullsized":"2024-09-21T05:45:14.256613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259518+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:45:14.259518+0000","last_peered":"2024-09-21T05:45:14.259518+0000","last_clean":"2024-09-21T05:45:14.259518+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:45:14.259518+0000","last_undegraded":"2024-09-21T05:45:14.259518+0000","last_fullsized":"2024-09-21T05:45:14.259518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:41.300480+0000","last_change":"2024-09-21T05:45:14.267608+0000","last_active":"2024-09-21T05:45:41.300480+0000","last_peered":"2024-09-21T05:45:41.300480+0000","last_clean":"2024-09-21T05:45:41.300480+0000","last_became_active":"2024-09-21T05:45:14.260846+0000","last_became_peered":"2024-09-21T05:45:14.260846+0000","last_unstale":"2024-09-21T05:45:41.300480+0000","last_undegraded":"2024-09-21T05:45:41.300480+0000","last_fullsized":"2024-09-21T05:45:41.300480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:29.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253038+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:45:14.253038+0000","last_peered":"2024-09-21T05:45:14.253038+0000","last_clean":"2024-09-21T05:45:14.253038+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:45:14.253038+0000","last_undegraded":"2024-09-21T05:45:14.253038+0000","last_fullsized":"2024-09-21T05:45:14.253038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.275469+0000","last_change":"2024-09-21T05:45:14.275384+0000","last_active":"2024-09-21T05:45:14.275469+0000","last_peered":"2024-09-21T05:45:14.275469+0000","last_clean":"2024-09-21T05:45:14.275469+0000","last_became_active":"2024-09-21T05:45:14.265194+0000","last_became_peered":"2024-09-21T05:45:14.265194+0000","last_unstale":"2024-09-21T05:45:14.275469+0000","last_undegraded":"2024-09-21T05:45:14.275469+0000","last_fullsized":"2024-09-21T05:45:14.275469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:37:24.227108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252190+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:45:14.252190+0000","last_peered":"2024-09-21T05:45:14.252190+0000","last_clean":"2024-09-21T05:45:14.252190+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:45:14.252190+0000","last_undegraded":"2024-09-21T05:45:14.252190+0000","last_fullsized":"2024-09-21T05:45:14.252190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321535+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:45:14.321535+0000","last_peered":"2024-09-21T05:45:14.321535+0000","last_clean":"2024-09-21T05:45:14.321535+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:45:14.321535+0000","last_undegraded":"2024-09-21T05:45:14.321535+0000","last_fullsized":"2024-09-21T05:45:14.321535+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.278537+0000","last_change":"2024-09-21T05:45:14.278120+0000","last_active":"2024-09-21T05:45:14.278537+0000","last_peered":"2024-09-21T05:45:14.278537+0000","last_clean":"2024-09-21T05:45:14.278537+0000","last_became_active":"2024-09-21T05:45:14.263713+0000","last_became_peered":"2024-09-21T05:45:14.263713+0000","last_unstale":"2024-09-21T05:45:14.278537+0000","last_undegraded":"2024-09-21T05:45:14.278537+0000","last_fullsized":"2024-09-21T05:45:14.278537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:16.259132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255257+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:45:14.255257+0000","last_peered":"2024-09-21T05:45:14.255257+0000","last_clean":"2024-09-21T05:45:14.255257+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:45:14.255257+0000","last_undegraded":"2024-09-21T05:45:14.255257+0000","last_fullsized":"2024-09-21T05:45:14.255257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"71'4","reported_seq":129,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.134418+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:45:42.134418+0000","last_peered":"2024-09-21T05:45:42.134418+0000","last_clean":"2024-09-21T05:45:42.134418+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:45:42.134418+0000","last_undegraded":"2024-09-21T05:45:42.134418+0000","last_fullsized":"2024-09-21T05:45:42.134418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262892+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:45:14.262892+0000","last_peered":"2024-09-21T05:45:14.262892+0000","last_clean":"2024-09-21T05:45:14.262892+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:45:14.262892+0000","last_undegraded":"2024-09-21T05:45:14.262892+0000","last_fullsized":"2024-09-21T05:45:14.262892+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.270877+0000","last_change":"2024-09-21T05:45:14.270775+0000","last_active":"2024-09-21T05:45:14.270877+0000","last_peered":"2024-09-21T05:45:14.270877+0000","last_clean":"2024-09-21T05:45:14.270877+0000","last_became_active":"2024-09-21T05:45:14.263657+0000","last_became_peered":"2024-09-21T05:45:14.263657+0000","last_unstale":"2024-09-21T05:45:14.270877+0000","last_undegraded":"2024-09-21T05:45:14.270877+0000","last_fullsized":"2024-09-21T05:45:14.270877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:27.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260628+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:45:14.260628+0000","last_peered":"2024-09-21T05:45:14.260628+0000","last_clean":"2024-09-21T05:45:14.260628+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:45:14.260628+0000","last_undegraded":"2024-09-21T05:45:14.260628+0000","last_fullsized":"2024-09-21T05:45:14.260628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253308+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:45:14.253308+0000","last_peered":"2024-09-21T05:45:14.253308+0000","last_clean":"2024-09-21T05:45:14.253308+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:45:14.253308+0000","last_undegraded":"2024-09-21T05:45:14.253308+0000","last_fullsized":"2024-09-21T05:45:14.253308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262448+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:45:14.262448+0000","last_peered":"2024-09-21T05:45:14.262448+0000","last_clean":"2024-09-21T05:45:14.262448+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:45:14.262448+0000","last_undegraded":"2024-09-21T05:45:14.262448+0000","last_fullsized":"2024-09-21T05:45:14.262448+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"71'30","reported_seq":163,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.301287+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:45:42.301287+0000","last_peered":"2024-09-21T05:45:42.301287+0000","last_clean":"2024-09-21T05:45:42.301287+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:45:42.301287+0000","last_undegraded":"2024-09-21T05:45:42.301287+0000","last_fullsized":"2024-09-21T05:45:42.301287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64757,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":117,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258720+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:45:14.258720+0000","last_peered":"2024-09-21T05:45:14.258720+0000","last_clean":"2024-09-21T05:45:14.258720+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:45:14.258720+0000","last_undegraded":"2024-09-21T05:45:14.258720+0000","last_fullsized":"2024-09-21T05:45:14.258720+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256407+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:45:14.256407+0000","last_peered":"2024-09-21T05:45:14.256407+0000","last_clean":"2024-09-21T05:45:14.256407+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:45:14.256407+0000","last_undegraded":"2024-09-21T05:45:14.256407+0000","last_fullsized":"2024-09-21T05:45:14.256407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252219+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:45:14.252219+0000","last_peered":"2024-09-21T05:45:14.252219+0000","last_clean":"2024-09-21T05:45:14.252219+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:45:14.252219+0000","last_undegraded":"2024-09-21T05:45:14.252219+0000","last_fullsized":"2024-09-21T05:45:14.252219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253000+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:45:14.253000+0000","last_peered":"2024-09-21T05:45:14.253000+0000","last_clean":"2024-09-21T05:45:14.253000+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:45:14.253000+0000","last_undegraded":"2024-09-21T05:45:14.253000+0000","last_fullsized":"2024-09-21T05:45:14.253000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256434+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:45:14.256434+0000","last_peered":"2024-09-21T05:45:14.256434+0000","last_clean":"2024-09-21T05:45:14.256434+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:45:14.256434+0000","last_undegraded":"2024-09-21T05:45:14.256434+0000","last_fullsized":"2024-09-21T05:45:14.256434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":77,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:37.124091+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:45:37.124091+0000","last_peered":"2024-09-21T05:45:37.124091+0000","last_clean":"2024-09-21T05:45:37.124091+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:45:37.124091+0000","last_undegraded":"2024-09-21T05:45:37.124091+0000","last_fullsized":"2024-09-21T05:45:37.124091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255842+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:45:14.255842+0000","last_peered":"2024-09-21T05:45:14.255842+0000","last_clean":"2024-09-21T05:45:14.255842+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:45:14.255842+0000","last_undegraded":"2024-09-21T05:45:14.255842+0000","last_fullsized":"2024-09-21T05:45:14.255842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":41,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258397+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:45:14.258397+0000","last_peered":"2024-09-21T05:45:14.258397+0000","last_clean":"2024-09-21T05:45:14.258397+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:45:14.258397+0000","last_undegraded":"2024-09-21T05:45:14.258397+0000","last_fullsized":"2024-09-21T05:45:14.258397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252913+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:45:14.252913+0000","last_peered":"2024-09-21T05:45:14.252913+0000","last_clean":"2024-09-21T05:45:14.252913+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:45:14.252913+0000","last_undegraded":"2024-09-21T05:45:14.252913+0000","last_fullsized":"2024-09-21T05:45:14.252913+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259227+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:45:14.259227+0000","last_peered":"2024-09-21T05:45:14.259227+0000","last_clean":"2024-09-21T05:45:14.259227+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:45:14.259227+0000","last_undegraded":"2024-09-21T05:45:14.259227+0000","last_fullsized":"2024-09-21T05:45:14.259227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263229+0000","last_change":"2024-09-21T05:45:14.263088+0000","last_active":"2024-09-21T05:45:14.263229+0000","last_peered":"2024-09-21T05:45:14.263229+0000","last_clean":"2024-09-21T05:45:14.263229+0000","last_became_active":"2024-09-21T05:45:14.262643+0000","last_became_peered":"2024-09-21T05:45:14.262643+0000","last_unstale":"2024-09-21T05:45:14.263229+0000","last_undegraded":"2024-09-21T05:45:14.263229+0000","last_fullsized":"2024-09-21T05:45:14.263229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:05.991971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252502+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:45:14.252502+0000","last_peered":"2024-09-21T05:45:14.252502+0000","last_clean":"2024-09-21T05:45:14.252502+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:45:14.252502+0000","last_undegraded":"2024-09-21T05:45:14.252502+0000","last_fullsized":"2024-09-21T05:45:14.252502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256412+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:45:14.256412+0000","last_peered":"2024-09-21T05:45:14.256412+0000","last_clean":"2024-09-21T05:45:14.256412+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:45:14.256412+0000","last_undegraded":"2024-09-21T05:45:14.256412+0000","last_fullsized":"2024-09-21T05:45:14.256412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253487+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:45:14.253487+0000","last_peered":"2024-09-21T05:45:14.253487+0000","last_clean":"2024-09-21T05:45:14.253487+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:45:14.253487+0000","last_undegraded":"2024-09-21T05:45:14.253487+0000","last_fullsized":"2024-09-21T05:45:14.253487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"71'1","reported_seq":19,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256351+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:45:14.256351+0000","last_peered":"2024-09-21T05:45:14.256351+0000","last_clean":"2024-09-21T05:45:14.256351+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:45:14.256351+0000","last_undegraded":"2024-09-21T05:45:14.256351+0000","last_fullsized":"2024-09-21T05:45:14.256351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"71'6","reported_seq":82,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:40.131988+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:45:40.131988+0000","last_peered":"2024-09-21T05:45:40.131988+0000","last_clean":"2024-09-21T05:45:40.131988+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:45:40.131988+0000","last_undegraded":"2024-09-21T05:45:40.131988+0000","last_fullsized":"2024-09-21T05:45:40.131988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256425+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:45:14.256425+0000","last_peered":"2024-09-21T05:45:14.256425+0000","last_clean":"2024-09-21T05:45:14.256425+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:45:14.256425+0000","last_undegraded":"2024-09-21T05:45:14.256425+0000","last_fullsized":"2024-09-21T05:45:14.256425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257010+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:45:14.257010+0000","last_peered":"2024-09-21T05:45:14.257010+0000","last_clean":"2024-09-21T05:45:14.257010+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:45:14.257010+0000","last_undegraded":"2024-09-21T05:45:14.257010+0000","last_fullsized":"2024-09-21T05:45:14.257010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":32,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253471+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:45:14.253471+0000","last_peered":"2024-09-21T05:45:14.253471+0000","last_clean":"2024-09-21T05:45:14.253471+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:45:14.253471+0000","last_undegraded":"2024-09-21T05:45:14.253471+0000","last_fullsized":"2024-09-21T05:45:14.253471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321773+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:45:14.321773+0000","last_peered":"2024-09-21T05:45:14.321773+0000","last_clean":"2024-09-21T05:45:14.321773+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:45:14.321773+0000","last_undegraded":"2024-09-21T05:45:14.321773+0000","last_fullsized":"2024-09-21T05:45:14.321773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256020+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:45:14.256020+0000","last_peered":"2024-09-21T05:45:14.256020+0000","last_clean":"2024-09-21T05:45:14.256020+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:45:14.256020+0000","last_undegraded":"2024-09-21T05:45:14.256020+0000","last_fullsized":"2024-09-21T05:45:14.256020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252050+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:45:14.252050+0000","last_peered":"2024-09-21T05:45:14.252050+0000","last_clean":"2024-09-21T05:45:14.252050+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:45:14.252050+0000","last_undegraded":"2024-09-21T05:45:14.252050+0000","last_fullsized":"2024-09-21T05:45:14.252050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":79,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:40.127736+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:45:40.127736+0000","last_peered":"2024-09-21T05:45:40.127736+0000","last_clean":"2024-09-21T05:45:40.127736+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:45:40.127736+0000","last_undegraded":"2024-09-21T05:45:40.127736+0000","last_fullsized":"2024-09-21T05:45:40.127736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252904+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:45:14.252904+0000","last_peered":"2024-09-21T05:45:14.252904+0000","last_clean":"2024-09-21T05:45:14.252904+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:45:14.252904+0000","last_undegraded":"2024-09-21T05:45:14.252904+0000","last_fullsized":"2024-09-21T05:45:14.252904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":78,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:39.142480+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:45:39.142480+0000","last_peered":"2024-09-21T05:45:39.142480+0000","last_clean":"2024-09-21T05:45:39.142480+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:45:39.142480+0000","last_undegraded":"2024-09-21T05:45:39.142480+0000","last_fullsized":"2024-09-21T05:45:39.142480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255021+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:45:14.255021+0000","last_peered":"2024-09-21T05:45:14.255021+0000","last_clean":"2024-09-21T05:45:14.255021+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:45:14.255021+0000","last_undegraded":"2024-09-21T05:45:14.255021+0000","last_fullsized":"2024-09-21T05:45:14.255021+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255492+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:45:14.255492+0000","last_peered":"2024-09-21T05:45:14.255492+0000","last_clean":"2024-09-21T05:45:14.255492+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:45:14.255492+0000","last_undegraded":"2024-09-21T05:45:14.255492+0000","last_fullsized":"2024-09-21T05:45:14.255492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"71'1","reported_seq":19,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254977+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:45:14.254977+0000","last_peered":"2024-09-21T05:45:14.254977+0000","last_clean":"2024-09-21T05:45:14.254977+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:45:14.254977+0000","last_undegraded":"2024-09-21T05:45:14.254977+0000","last_fullsized":"2024-09-21T05:45:14.254977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.269448+0000","last_change":"2024-09-21T05:45:14.268855+0000","last_active":"2024-09-21T05:45:14.269448+0000","last_peered":"2024-09-21T05:45:14.269448+0000","last_clean":"2024-09-21T05:45:14.269448+0000","last_became_active":"2024-09-21T05:45:14.261558+0000","last_became_peered":"2024-09-21T05:45:14.261558+0000","last_unstale":"2024-09-21T05:45:14.269448+0000","last_undegraded":"2024-09-21T05:45:14.269448+0000","last_fullsized":"2024-09-21T05:45:14.269448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:57.978428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252449+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:45:14.252449+0000","last_peered":"2024-09-21T05:45:14.252449+0000","last_clean":"2024-09-21T05:45:14.252449+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:45:14.252449+0000","last_undegraded":"2024-09-21T05:45:14.252449+0000","last_fullsized":"2024-09-21T05:45:14.252449+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"71'14","reported_seq":37,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255620+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:14.255620+0000","last_peered":"2024-09-21T05:45:14.255620+0000","last_clean":"2024-09-21T05:45:14.255620+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:14.255620+0000","last_undegraded":"2024-09-21T05:45:14.255620+0000","last_fullsized":"2024-09-21T05:45:14.255620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.261566+0000","last_change":"2024-09-21T05:45:14.261447+0000","last_active":"2024-09-21T05:45:14.261566+0000","last_peered":"2024-09-21T05:45:14.261566+0000","last_clean":"2024-09-21T05:45:14.261566+0000","last_became_active":"2024-09-21T05:45:14.261072+0000","last_became_peered":"2024-09-21T05:45:14.261072+0000","last_unstale":"2024-09-21T05:45:14.261566+0000","last_undegraded":"2024-09-21T05:45:14.261566+0000","last_fullsized":"2024-09-21T05:45:14.261566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:46.148923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254328+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:45:14.254328+0000","last_peered":"2024-09-21T05:45:14.254328+0000","last_clean":"2024-09-21T05:45:14.254328+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:45:14.254328+0000","last_undegraded":"2024-09-21T05:45:14.254328+0000","last_fullsized":"2024-09-21T05:45:14.254328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260424+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:45:14.260424+0000","last_peered":"2024-09-21T05:45:14.260424+0000","last_clean":"2024-09-21T05:45:14.260424+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:45:14.260424+0000","last_undegraded":"2024-09-21T05:45:14.260424+0000","last_fullsized":"2024-09-21T05:45:14.260424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262195+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:45:14.262195+0000","last_peered":"2024-09-21T05:45:14.262195+0000","last_clean":"2024-09-21T05:45:14.262195+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:45:14.262195+0000","last_undegraded":"2024-09-21T05:45:14.262195+0000","last_fullsized":"2024-09-21T05:45:14.262195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258557+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:45:14.258557+0000","last_peered":"2024-09-21T05:45:14.258557+0000","last_clean":"2024-09-21T05:45:14.258557+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:45:14.258557+0000","last_undegraded":"2024-09-21T05:45:14.258557+0000","last_fullsized":"2024-09-21T05:45:14.258557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321694+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:45:14.321694+0000","last_peered":"2024-09-21T05:45:14.321694+0000","last_clean":"2024-09-21T05:45:14.321694+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:45:14.321694+0000","last_undegraded":"2024-09-21T05:45:14.321694+0000","last_fullsized":"2024-09-21T05:45:14.321694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256737+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:45:14.256737+0000","last_peered":"2024-09-21T05:45:14.256737+0000","last_clean":"2024-09-21T05:45:14.256737+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:45:14.256737+0000","last_undegraded":"2024-09-21T05:45:14.256737+0000","last_fullsized":"2024-09-21T05:45:14.256737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256606+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:45:14.256606+0000","last_peered":"2024-09-21T05:45:14.256606+0000","last_clean":"2024-09-21T05:45:14.256606+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:45:14.256606+0000","last_undegraded":"2024-09-21T05:45:14.256606+0000","last_fullsized":"2024-09-21T05:45:14.256606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321933+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:45:14.321933+0000","last_peered":"2024-09-21T05:45:14.321933+0000","last_clean":"2024-09-21T05:45:14.321933+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:45:14.321933+0000","last_undegraded":"2024-09-21T05:45:14.321933+0000","last_fullsized":"2024-09-21T05:45:14.321933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258860+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:45:14.258860+0000","last_peered":"2024-09-21T05:45:14.258860+0000","last_clean":"2024-09-21T05:45:14.258860+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:45:14.258860+0000","last_undegraded":"2024-09-21T05:45:14.258860+0000","last_fullsized":"2024-09-21T05:45:14.258860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256438+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:45:14.256438+0000","last_peered":"2024-09-21T05:45:14.256438+0000","last_clean":"2024-09-21T05:45:14.256438+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:45:14.256438+0000","last_undegraded":"2024-09-21T05:45:14.256438+0000","last_fullsized":"2024-09-21T05:45:14.256438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263778+0000","last_change":"2024-09-21T05:45:14.263694+0000","last_active":"2024-09-21T05:45:14.263778+0000","last_peered":"2024-09-21T05:45:14.263778+0000","last_clean":"2024-09-21T05:45:14.263778+0000","last_became_active":"2024-09-21T05:45:14.263394+0000","last_became_peered":"2024-09-21T05:45:14.263394+0000","last_unstale":"2024-09-21T05:45:14.263778+0000","last_undegraded":"2024-09-21T05:45:14.263778+0000","last_fullsized":"2024-09-21T05:45:14.263778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:59.214553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260360+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:45:14.260360+0000","last_peered":"2024-09-21T05:45:14.260360+0000","last_clean":"2024-09-21T05:45:14.260360+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:45:14.260360+0000","last_undegraded":"2024-09-21T05:45:14.260360+0000","last_fullsized":"2024-09-21T05:45:14.260360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262259+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:45:14.262259+0000","last_peered":"2024-09-21T05:45:14.262259+0000","last_clean":"2024-09-21T05:45:14.262259+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:45:14.262259+0000","last_undegraded":"2024-09-21T05:45:14.262259+0000","last_fullsized":"2024-09-21T05:45:14.262259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":36,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258852+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:45:14.258852+0000","last_peered":"2024-09-21T05:45:14.258852+0000","last_clean":"2024-09-21T05:45:14.258852+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:45:14.258852+0000","last_undegraded":"2024-09-21T05:45:14.258852+0000","last_fullsized":"2024-09-21T05:45:14.258852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260516+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:45:14.260516+0000","last_peered":"2024-09-21T05:45:14.260516+0000","last_clean":"2024-09-21T05:45:14.260516+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:45:14.260516+0000","last_undegraded":"2024-09-21T05:45:14.260516+0000","last_fullsized":"2024-09-21T05:45:14.260516+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322148+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:45:14.322148+0000","last_peered":"2024-09-21T05:45:14.322148+0000","last_clean":"2024-09-21T05:45:14.322148+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:45:14.322148+0000","last_undegraded":"2024-09-21T05:45:14.322148+0000","last_fullsized":"2024-09-21T05:45:14.322148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322040+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:45:14.322040+0000","last_peered":"2024-09-21T05:45:14.322040+0000","last_clean":"2024-09-21T05:45:14.322040+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:45:14.322040+0000","last_undegraded":"2024-09-21T05:45:14.322040+0000","last_fullsized":"2024-09-21T05:45:14.322040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":84,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:37.139041+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:45:37.139041+0000","last_peered":"2024-09-21T05:45:37.139041+0000","last_clean":"2024-09-21T05:45:37.139041+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:37.139041+0000","last_undegraded":"2024-09-21T05:45:37.139041+0000","last_fullsized":"2024-09-21T05:45:37.139041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71570,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":523,"num_read_kb":345,"num_write":241,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1316,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874266,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":819200,"data_stored":399975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037791,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32668,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712484,"statfs":{"total":95995035648,"available":95961583616,"internally_reserved":0,"allocated":1511424,"data_stored":1054676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201315,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36164,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708988,"statfs":{"total":95995035648,"available":95958003712,"internally_reserved":0,"allocated":831488,"data_stored":398820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":819200,"data_stored":399086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36744,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708408,"statfs":{"total":95995035648,"available":95957409792,"internally_reserved":0,"allocated":1425408,"data_stored":991325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724594,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":815104,"data_stored":398537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888119,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1433600,"data_stored":990421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018939,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":823296,"data_stored":399020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215168,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":831488,"data_stored":398362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":892928,"data_stored":464196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":880640,"data_stored":464207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":835584,"data_stored":400267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:45:44.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T05:45:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:45.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:45.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:45:45.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:45:45.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:45:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 -- 172.21.15.28:0/1379370868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860104a30 msgr2=0x7f2860104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 --2- 172.21.15.28:0/1379370868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860104a30 0x7f2860104e40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2850009ca0 tx=0x7f285002a8e0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 -- 172.21.15.28:0/1379370868 shutdown_connections 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 --2- 172.21.15.28:0/1379370868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 0x7f2860110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 --2- 172.21.15.28:0/1379370868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860105380 0x7f286010d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 --2- 172.21.15.28:0/1379370868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860104a30 0x7f2860104e40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 -- 172.21.15.28:0/1379370868 >> 172.21.15.28:0/1379370868 conn(0x7f2860079760 msgr2=0x7f2860079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 -- 172.21.15.28:0/1379370868 shutdown_connections 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 -- 172.21.15.28:0/1379370868 wait complete. 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.667+0000 7f2865844640 1 Processor -- start 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 -- start start 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860104a30 0x7f2860102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 0x7f28601031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2860113b80 con 0x7f2860105380 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2860113a00 con 0x7f2860104a30 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f2865844640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2860113d00 con 0x7f286010de60 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53742/0 (socket says 172.21.15.28:53742) 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 -- 172.21.15.28:0/1257414870 learned_addr learned my addr 172.21.15.28:0/1257414870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285f7fe640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 0x7f28601031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285effd640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860104a30 0x7f2860102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 -- 172.21.15.28:0/1257414870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 msgr2=0x7f28601031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 0x7f28601031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 -- 172.21.15.28:0/1257414870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860104a30 msgr2=0x7f2860102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860104a30 0x7f2860102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 -- 172.21.15.28:0/1257414870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28601a8950 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.668+0000 7f285e7fc640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2854006fd0 tx=0x7f285400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.669+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854011830 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.669+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2854016070 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.669+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28601a8c40 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.669+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854010e00 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.669+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28601a9180 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.670+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286006c1f0 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.671+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f285401b460 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.672+0000 7f2864842640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 0x7f28480b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.673+0000 7f285effd640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 0x7f28480b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.673+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f28540e7130 con 0x7f2860105380 2024-09-21T05:45:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.673+0000 7f285effd640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 0x7f28480b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2850000f80 tx=0x7f28500093e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:45.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.676+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28540ad760 con 0x7f2860105380 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:45:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:45:45.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.866+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2860063800 con 0x7f28480b65f0 2024-09-21T05:45:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.872+0000 7f2864842640 1 -- 172.21.15.28:0/1257414870 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258757 (secure 0 0 0) 0x7f2860063800 con 0x7f28480b65f0 2024-09-21T05:45:45.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 msgr2=0x7f28480b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 0x7f28480b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2850000f80 tx=0x7f28500093e0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 msgr2=0x7f2860102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2854006fd0 tx=0x7f285400b8e0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 shutdown_connections 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f286010de60 0x7f28601031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28480b65f0 0x7f28480b8b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2860105380 0x7f2860102cb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 --2- 172.21.15.28:0/1257414870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2860104a30 0x7f2860102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 >> 172.21.15.28:0/1257414870 conn(0x7f2860079760 msgr2=0x7f28601009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 shutdown_connections 2024-09-21T05:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:45.876+0000 7f2865844640 1 -- 172.21.15.28:0/1257414870 wait complete. 2024-09-21T05:45:46.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:46 smithi028 ceph-mon[31227]: pgmap v287: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:46.494 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-21T05:45:44.727650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662172,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":614,"num_read_kb":419,"num_write":366,"num_write_kb":2290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2953156,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385928,"kb_used_data":11640,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555896,"statfs":{"total":1151940427776,"available":1151545237504,"internally_reserved":0,"allocated":11919360,"data_stored":6758892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003596"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256259+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:45:14.256259+0000","last_peered":"2024-09-21T05:45:14.256259+0000","last_clean":"2024-09-21T05:45:14.256259+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:45:14.256259+0000","last_undegraded":"2024-09-21T05:45:14.256259+0000","last_fullsized":"2024-09-21T05:45:14.256259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260266+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:45:14.260266+0000","last_peered":"2024-09-21T05:45:14.260266+0000","last_clean":"2024-09-21T05:45:14.260266+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:45:14.260266+0000","last_undegraded":"2024-09-21T05:45:14.260266+0000","last_fullsized":"2024-09-21T05:45:14.260266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260390+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:45:14.260390+0000","last_peered":"2024-09-21T05:45:14.260390+0000","last_clean":"2024-09-21T05:45:14.260390+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:45:14.260390+0000","last_undegraded":"2024-09-21T05:45:14.260390+0000","last_fullsized":"2024-09-21T05:45:14.260390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253684+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:45:14.253684+0000","last_peered":"2024-09-21T05:45:14.253684+0000","last_clean":"2024-09-21T05:45:14.253684+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:45:14.253684+0000","last_undegraded":"2024-09-21T05:45:14.253684+0000","last_fullsized":"2024-09-21T05:45:14.253684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258728+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:45:14.258728+0000","last_peered":"2024-09-21T05:45:14.258728+0000","last_clean":"2024-09-21T05:45:14.258728+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:45:14.258728+0000","last_undegraded":"2024-09-21T05:45:14.258728+0000","last_fullsized":"2024-09-21T05:45:14.258728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259120+0000","last_change":"2024-09-21T05:45:14.258964+0000","last_active":"2024-09-21T05:45:14.259120+0000","last_peered":"2024-09-21T05:45:14.259120+0000","last_clean":"2024-09-21T05:45:14.259120+0000","last_became_active":"2024-09-21T05:45:14.257864+0000","last_became_peered":"2024-09-21T05:45:14.257864+0000","last_unstale":"2024-09-21T05:45:14.259120+0000","last_undegraded":"2024-09-21T05:45:14.259120+0000","last_fullsized":"2024-09-21T05:45:14.259120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.693452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256026+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:45:14.256026+0000","last_peered":"2024-09-21T05:45:14.256026+0000","last_clean":"2024-09-21T05:45:14.256026+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:45:14.256026+0000","last_undegraded":"2024-09-21T05:45:14.256026+0000","last_fullsized":"2024-09-21T05:45:14.256026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321491+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:45:14.321491+0000","last_peered":"2024-09-21T05:45:14.321491+0000","last_clean":"2024-09-21T05:45:14.321491+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:45:14.321491+0000","last_undegraded":"2024-09-21T05:45:14.321491+0000","last_fullsized":"2024-09-21T05:45:14.321491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262607+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:45:14.262607+0000","last_peered":"2024-09-21T05:45:14.262607+0000","last_clean":"2024-09-21T05:45:14.262607+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:45:14.262607+0000","last_undegraded":"2024-09-21T05:45:14.262607+0000","last_fullsized":"2024-09-21T05:45:14.262607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255408+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:45:14.255408+0000","last_peered":"2024-09-21T05:45:14.255408+0000","last_clean":"2024-09-21T05:45:14.255408+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:45:14.255408+0000","last_undegraded":"2024-09-21T05:45:14.255408+0000","last_fullsized":"2024-09-21T05:45:14.255408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256306+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:45:14.256306+0000","last_peered":"2024-09-21T05:45:14.256306+0000","last_clean":"2024-09-21T05:45:14.256306+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:45:14.256306+0000","last_undegraded":"2024-09-21T05:45:14.256306+0000","last_fullsized":"2024-09-21T05:45:14.256306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253789+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:45:14.253789+0000","last_peered":"2024-09-21T05:45:14.253789+0000","last_clean":"2024-09-21T05:45:14.253789+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:45:14.253789+0000","last_undegraded":"2024-09-21T05:45:14.253789+0000","last_fullsized":"2024-09-21T05:45:14.253789+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252491+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:45:14.252491+0000","last_peered":"2024-09-21T05:45:14.252491+0000","last_clean":"2024-09-21T05:45:14.252491+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:45:14.252491+0000","last_undegraded":"2024-09-21T05:45:14.252491+0000","last_fullsized":"2024-09-21T05:45:14.252491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260165+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:45:14.260165+0000","last_peered":"2024-09-21T05:45:14.260165+0000","last_clean":"2024-09-21T05:45:14.260165+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:45:14.260165+0000","last_undegraded":"2024-09-21T05:45:14.260165+0000","last_fullsized":"2024-09-21T05:45:14.260165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260121+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:45:14.260121+0000","last_peered":"2024-09-21T05:45:14.260121+0000","last_clean":"2024-09-21T05:45:14.260121+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:45:14.260121+0000","last_undegraded":"2024-09-21T05:45:14.260121+0000","last_fullsized":"2024-09-21T05:45:14.260121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262111+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:45:14.262111+0000","last_peered":"2024-09-21T05:45:14.262111+0000","last_clean":"2024-09-21T05:45:14.262111+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:45:14.262111+0000","last_undegraded":"2024-09-21T05:45:14.262111+0000","last_fullsized":"2024-09-21T05:45:14.262111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256963+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:45:14.256963+0000","last_peered":"2024-09-21T05:45:14.256963+0000","last_clean":"2024-09-21T05:45:14.256963+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:45:14.256963+0000","last_undegraded":"2024-09-21T05:45:14.256963+0000","last_fullsized":"2024-09-21T05:45:14.256963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255456+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:45:14.255456+0000","last_peered":"2024-09-21T05:45:14.255456+0000","last_clean":"2024-09-21T05:45:14.255456+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:45:14.255456+0000","last_undegraded":"2024-09-21T05:45:14.255456+0000","last_fullsized":"2024-09-21T05:45:14.255456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256579+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:45:14.256579+0000","last_peered":"2024-09-21T05:45:14.256579+0000","last_clean":"2024-09-21T05:45:14.256579+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:45:14.256579+0000","last_undegraded":"2024-09-21T05:45:14.256579+0000","last_fullsized":"2024-09-21T05:45:14.256579+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263278+0000","last_change":"2024-09-21T05:45:14.263089+0000","last_active":"2024-09-21T05:45:14.263278+0000","last_peered":"2024-09-21T05:45:14.263278+0000","last_clean":"2024-09-21T05:45:14.263278+0000","last_became_active":"2024-09-21T05:45:14.262626+0000","last_became_peered":"2024-09-21T05:45:14.262626+0000","last_unstale":"2024-09-21T05:45:14.263278+0000","last_undegraded":"2024-09-21T05:45:14.263278+0000","last_fullsized":"2024-09-21T05:45:14.263278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:34.442511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252627+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:45:14.252627+0000","last_peered":"2024-09-21T05:45:14.252627+0000","last_clean":"2024-09-21T05:45:14.252627+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:45:14.252627+0000","last_undegraded":"2024-09-21T05:45:14.252627+0000","last_fullsized":"2024-09-21T05:45:14.252627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262734+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:45:14.262734+0000","last_peered":"2024-09-21T05:45:14.262734+0000","last_clean":"2024-09-21T05:45:14.262734+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:45:14.262734+0000","last_undegraded":"2024-09-21T05:45:14.262734+0000","last_fullsized":"2024-09-21T05:45:14.262734+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321494+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:45:14.321494+0000","last_peered":"2024-09-21T05:45:14.321494+0000","last_clean":"2024-09-21T05:45:14.321494+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:45:14.321494+0000","last_undegraded":"2024-09-21T05:45:14.321494+0000","last_fullsized":"2024-09-21T05:45:14.321494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252852+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:45:14.252852+0000","last_peered":"2024-09-21T05:45:14.252852+0000","last_clean":"2024-09-21T05:45:14.252852+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:45:14.252852+0000","last_undegraded":"2024-09-21T05:45:14.252852+0000","last_fullsized":"2024-09-21T05:45:14.252852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321687+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:45:14.321687+0000","last_peered":"2024-09-21T05:45:14.321687+0000","last_clean":"2024-09-21T05:45:14.321687+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:45:14.321687+0000","last_undegraded":"2024-09-21T05:45:14.321687+0000","last_fullsized":"2024-09-21T05:45:14.321687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257077+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:45:14.257077+0000","last_peered":"2024-09-21T05:45:14.257077+0000","last_clean":"2024-09-21T05:45:14.257077+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:45:14.257077+0000","last_undegraded":"2024-09-21T05:45:14.257077+0000","last_fullsized":"2024-09-21T05:45:14.257077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256239+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:45:14.256239+0000","last_peered":"2024-09-21T05:45:14.256239+0000","last_clean":"2024-09-21T05:45:14.256239+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:45:14.256239+0000","last_undegraded":"2024-09-21T05:45:14.256239+0000","last_fullsized":"2024-09-21T05:45:14.256239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252494+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:45:14.252494+0000","last_peered":"2024-09-21T05:45:14.252494+0000","last_clean":"2024-09-21T05:45:14.252494+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:45:14.252494+0000","last_undegraded":"2024-09-21T05:45:14.252494+0000","last_fullsized":"2024-09-21T05:45:14.252494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260469+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:45:14.260469+0000","last_peered":"2024-09-21T05:45:14.260469+0000","last_clean":"2024-09-21T05:45:14.260469+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:45:14.260469+0000","last_undegraded":"2024-09-21T05:45:14.260469+0000","last_fullsized":"2024-09-21T05:45:14.260469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256487+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:45:14.256487+0000","last_peered":"2024-09-21T05:45:14.256487+0000","last_clean":"2024-09-21T05:45:14.256487+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:45:14.256487+0000","last_undegraded":"2024-09-21T05:45:14.256487+0000","last_fullsized":"2024-09-21T05:45:14.256487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256369+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:45:14.256369+0000","last_peered":"2024-09-21T05:45:14.256369+0000","last_clean":"2024-09-21T05:45:14.256369+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:45:14.256369+0000","last_undegraded":"2024-09-21T05:45:14.256369+0000","last_fullsized":"2024-09-21T05:45:14.256369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256613+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:45:14.256613+0000","last_peered":"2024-09-21T05:45:14.256613+0000","last_clean":"2024-09-21T05:45:14.256613+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:45:14.256613+0000","last_undegraded":"2024-09-21T05:45:14.256613+0000","last_fullsized":"2024-09-21T05:45:14.256613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259518+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:45:14.259518+0000","last_peered":"2024-09-21T05:45:14.259518+0000","last_clean":"2024-09-21T05:45:14.259518+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:45:14.259518+0000","last_undegraded":"2024-09-21T05:45:14.259518+0000","last_fullsized":"2024-09-21T05:45:14.259518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:41.300480+0000","last_change":"2024-09-21T05:45:14.267608+0000","last_active":"2024-09-21T05:45:41.300480+0000","last_peered":"2024-09-21T05:45:41.300480+0000","last_clean":"2024-09-21T05:45:41.300480+0000","last_became_active":"2024-09-21T05:45:14.260846+0000","last_became_peered":"2024-09-21T05:45:14.260846+0000","last_unstale":"2024-09-21T05:45:41.300480+0000","last_undegraded":"2024-09-21T05:45:41.300480+0000","last_fullsized":"2024-09-21T05:45:41.300480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:29.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253038+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:45:14.253038+0000","last_peered":"2024-09-21T05:45:14.253038+0000","last_clean":"2024-09-21T05:45:14.253038+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:45:14.253038+0000","last_undegraded":"2024-09-21T05:45:14.253038+0000","last_fullsized":"2024-09-21T05:45:14.253038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.275469+0000","last_change":"2024-09-21T05:45:14.275384+0000","last_active":"2024-09-21T05:45:14.275469+0000","last_peered":"2024-09-21T05:45:14.275469+0000","last_clean":"2024-09-21T05:45:14.275469+0000","last_became_active":"2024-09-21T05:45:14.265194+0000","last_became_peered":"2024-09-21T05:45:14.265194+0000","last_unstale":"2024-09-21T05:45:14.275469+0000","last_undegraded":"2024-09-21T05:45:14.275469+0000","last_fullsized":"2024-09-21T05:45:14.275469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:37:24.227108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252190+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:45:14.252190+0000","last_peered":"2024-09-21T05:45:14.252190+0000","last_clean":"2024-09-21T05:45:14.252190+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:45:14.252190+0000","last_undegraded":"2024-09-21T05:45:14.252190+0000","last_fullsized":"2024-09-21T05:45:14.252190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321535+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:45:14.321535+0000","last_peered":"2024-09-21T05:45:14.321535+0000","last_clean":"2024-09-21T05:45:14.321535+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:45:14.321535+0000","last_undegraded":"2024-09-21T05:45:14.321535+0000","last_fullsized":"2024-09-21T05:45:14.321535+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.278537+0000","last_change":"2024-09-21T05:45:14.278120+0000","last_active":"2024-09-21T05:45:14.278537+0000","last_peered":"2024-09-21T05:45:14.278537+0000","last_clean":"2024-09-21T05:45:14.278537+0000","last_became_active":"2024-09-21T05:45:14.263713+0000","last_became_peered":"2024-09-21T05:45:14.263713+0000","last_unstale":"2024-09-21T05:45:14.278537+0000","last_undegraded":"2024-09-21T05:45:14.278537+0000","last_fullsized":"2024-09-21T05:45:14.278537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:16.259132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255257+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:45:14.255257+0000","last_peered":"2024-09-21T05:45:14.255257+0000","last_clean":"2024-09-21T05:45:14.255257+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:45:14.255257+0000","last_undegraded":"2024-09-21T05:45:14.255257+0000","last_fullsized":"2024-09-21T05:45:14.255257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"71'4","reported_seq":129,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.134418+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:45:42.134418+0000","last_peered":"2024-09-21T05:45:42.134418+0000","last_clean":"2024-09-21T05:45:42.134418+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:45:42.134418+0000","last_undegraded":"2024-09-21T05:45:42.134418+0000","last_fullsized":"2024-09-21T05:45:42.134418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262892+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:45:14.262892+0000","last_peered":"2024-09-21T05:45:14.262892+0000","last_clean":"2024-09-21T05:45:14.262892+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:45:14.262892+0000","last_undegraded":"2024-09-21T05:45:14.262892+0000","last_fullsized":"2024-09-21T05:45:14.262892+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.270877+0000","last_change":"2024-09-21T05:45:14.270775+0000","last_active":"2024-09-21T05:45:14.270877+0000","last_peered":"2024-09-21T05:45:14.270877+0000","last_clean":"2024-09-21T05:45:14.270877+0000","last_became_active":"2024-09-21T05:45:14.263657+0000","last_became_peered":"2024-09-21T05:45:14.263657+0000","last_unstale":"2024-09-21T05:45:14.270877+0000","last_undegraded":"2024-09-21T05:45:14.270877+0000","last_fullsized":"2024-09-21T05:45:14.270877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:27.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260628+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:45:14.260628+0000","last_peered":"2024-09-21T05:45:14.260628+0000","last_clean":"2024-09-21T05:45:14.260628+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:45:14.260628+0000","last_undegraded":"2024-09-21T05:45:14.260628+0000","last_fullsized":"2024-09-21T05:45:14.260628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253308+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:45:14.253308+0000","last_peered":"2024-09-21T05:45:14.253308+0000","last_clean":"2024-09-21T05:45:14.253308+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:45:14.253308+0000","last_undegraded":"2024-09-21T05:45:14.253308+0000","last_fullsized":"2024-09-21T05:45:14.253308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262448+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:45:14.262448+0000","last_peered":"2024-09-21T05:45:14.262448+0000","last_clean":"2024-09-21T05:45:14.262448+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:45:14.262448+0000","last_undegraded":"2024-09-21T05:45:14.262448+0000","last_fullsized":"2024-09-21T05:45:14.262448+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"71'30","reported_seq":163,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.301287+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:45:42.301287+0000","last_peered":"2024-09-21T05:45:42.301287+0000","last_clean":"2024-09-21T05:45:42.301287+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:45:42.301287+0000","last_undegraded":"2024-09-21T05:45:42.301287+0000","last_fullsized":"2024-09-21T05:45:42.301287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64757,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":117,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258720+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:45:14.258720+0000","last_peered":"2024-09-21T05:45:14.258720+0000","last_clean":"2024-09-21T05:45:14.258720+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:45:14.258720+0000","last_undegraded":"2024-09-21T05:45:14.258720+0000","last_fullsized":"2024-09-21T05:45:14.258720+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256407+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:45:14.256407+0000","last_peered":"2024-09-21T05:45:14.256407+0000","last_clean":"2024-09-21T05:45:14.256407+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:45:14.256407+0000","last_undegraded":"2024-09-21T05:45:14.256407+0000","last_fullsized":"2024-09-21T05:45:14.256407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252219+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:45:14.252219+0000","last_peered":"2024-09-21T05:45:14.252219+0000","last_clean":"2024-09-21T05:45:14.252219+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:45:14.252219+0000","last_undegraded":"2024-09-21T05:45:14.252219+0000","last_fullsized":"2024-09-21T05:45:14.252219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253000+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:45:14.253000+0000","last_peered":"2024-09-21T05:45:14.253000+0000","last_clean":"2024-09-21T05:45:14.253000+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:45:14.253000+0000","last_undegraded":"2024-09-21T05:45:14.253000+0000","last_fullsized":"2024-09-21T05:45:14.253000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256434+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:45:14.256434+0000","last_peered":"2024-09-21T05:45:14.256434+0000","last_clean":"2024-09-21T05:45:14.256434+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:45:14.256434+0000","last_undegraded":"2024-09-21T05:45:14.256434+0000","last_fullsized":"2024-09-21T05:45:14.256434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":82,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.132147+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:45:42.132147+0000","last_peered":"2024-09-21T05:45:42.132147+0000","last_clean":"2024-09-21T05:45:42.132147+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:45:42.132147+0000","last_undegraded":"2024-09-21T05:45:42.132147+0000","last_fullsized":"2024-09-21T05:45:42.132147+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255842+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:45:14.255842+0000","last_peered":"2024-09-21T05:45:14.255842+0000","last_clean":"2024-09-21T05:45:14.255842+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:45:14.255842+0000","last_undegraded":"2024-09-21T05:45:14.255842+0000","last_fullsized":"2024-09-21T05:45:14.255842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":41,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258397+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:45:14.258397+0000","last_peered":"2024-09-21T05:45:14.258397+0000","last_clean":"2024-09-21T05:45:14.258397+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:45:14.258397+0000","last_undegraded":"2024-09-21T05:45:14.258397+0000","last_fullsized":"2024-09-21T05:45:14.258397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252913+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:45:14.252913+0000","last_peered":"2024-09-21T05:45:14.252913+0000","last_clean":"2024-09-21T05:45:14.252913+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:45:14.252913+0000","last_undegraded":"2024-09-21T05:45:14.252913+0000","last_fullsized":"2024-09-21T05:45:14.252913+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259227+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:45:14.259227+0000","last_peered":"2024-09-21T05:45:14.259227+0000","last_clean":"2024-09-21T05:45:14.259227+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:45:14.259227+0000","last_undegraded":"2024-09-21T05:45:14.259227+0000","last_fullsized":"2024-09-21T05:45:14.259227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263229+0000","last_change":"2024-09-21T05:45:14.263088+0000","last_active":"2024-09-21T05:45:14.263229+0000","last_peered":"2024-09-21T05:45:14.263229+0000","last_clean":"2024-09-21T05:45:14.263229+0000","last_became_active":"2024-09-21T05:45:14.262643+0000","last_became_peered":"2024-09-21T05:45:14.262643+0000","last_unstale":"2024-09-21T05:45:14.263229+0000","last_undegraded":"2024-09-21T05:45:14.263229+0000","last_fullsized":"2024-09-21T05:45:14.263229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:05.991971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252502+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:45:14.252502+0000","last_peered":"2024-09-21T05:45:14.252502+0000","last_clean":"2024-09-21T05:45:14.252502+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:45:14.252502+0000","last_undegraded":"2024-09-21T05:45:14.252502+0000","last_fullsized":"2024-09-21T05:45:14.252502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256412+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:45:14.256412+0000","last_peered":"2024-09-21T05:45:14.256412+0000","last_clean":"2024-09-21T05:45:14.256412+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:45:14.256412+0000","last_undegraded":"2024-09-21T05:45:14.256412+0000","last_fullsized":"2024-09-21T05:45:14.256412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253487+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:45:14.253487+0000","last_peered":"2024-09-21T05:45:14.253487+0000","last_clean":"2024-09-21T05:45:14.253487+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:45:14.253487+0000","last_undegraded":"2024-09-21T05:45:14.253487+0000","last_fullsized":"2024-09-21T05:45:14.253487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"71'1","reported_seq":19,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256351+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:45:14.256351+0000","last_peered":"2024-09-21T05:45:14.256351+0000","last_clean":"2024-09-21T05:45:14.256351+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:45:14.256351+0000","last_undegraded":"2024-09-21T05:45:14.256351+0000","last_fullsized":"2024-09-21T05:45:14.256351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"71'6","reported_seq":82,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:40.131988+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:45:40.131988+0000","last_peered":"2024-09-21T05:45:40.131988+0000","last_clean":"2024-09-21T05:45:40.131988+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:45:40.131988+0000","last_undegraded":"2024-09-21T05:45:40.131988+0000","last_fullsized":"2024-09-21T05:45:40.131988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256425+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:45:14.256425+0000","last_peered":"2024-09-21T05:45:14.256425+0000","last_clean":"2024-09-21T05:45:14.256425+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:45:14.256425+0000","last_undegraded":"2024-09-21T05:45:14.256425+0000","last_fullsized":"2024-09-21T05:45:14.256425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257010+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:45:14.257010+0000","last_peered":"2024-09-21T05:45:14.257010+0000","last_clean":"2024-09-21T05:45:14.257010+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:45:14.257010+0000","last_undegraded":"2024-09-21T05:45:14.257010+0000","last_fullsized":"2024-09-21T05:45:14.257010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":32,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253471+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:45:14.253471+0000","last_peered":"2024-09-21T05:45:14.253471+0000","last_clean":"2024-09-21T05:45:14.253471+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:45:14.253471+0000","last_undegraded":"2024-09-21T05:45:14.253471+0000","last_fullsized":"2024-09-21T05:45:14.253471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321773+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:45:14.321773+0000","last_peered":"2024-09-21T05:45:14.321773+0000","last_clean":"2024-09-21T05:45:14.321773+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:45:14.321773+0000","last_undegraded":"2024-09-21T05:45:14.321773+0000","last_fullsized":"2024-09-21T05:45:14.321773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256020+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:45:14.256020+0000","last_peered":"2024-09-21T05:45:14.256020+0000","last_clean":"2024-09-21T05:45:14.256020+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:45:14.256020+0000","last_undegraded":"2024-09-21T05:45:14.256020+0000","last_fullsized":"2024-09-21T05:45:14.256020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252050+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:45:14.252050+0000","last_peered":"2024-09-21T05:45:14.252050+0000","last_clean":"2024-09-21T05:45:14.252050+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:45:14.252050+0000","last_undegraded":"2024-09-21T05:45:14.252050+0000","last_fullsized":"2024-09-21T05:45:14.252050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":79,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:40.127736+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:45:40.127736+0000","last_peered":"2024-09-21T05:45:40.127736+0000","last_clean":"2024-09-21T05:45:40.127736+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:45:40.127736+0000","last_undegraded":"2024-09-21T05:45:40.127736+0000","last_fullsized":"2024-09-21T05:45:40.127736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252904+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:45:14.252904+0000","last_peered":"2024-09-21T05:45:14.252904+0000","last_clean":"2024-09-21T05:45:14.252904+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:45:14.252904+0000","last_undegraded":"2024-09-21T05:45:14.252904+0000","last_fullsized":"2024-09-21T05:45:14.252904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":78,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:39.142480+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:45:39.142480+0000","last_peered":"2024-09-21T05:45:39.142480+0000","last_clean":"2024-09-21T05:45:39.142480+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:45:39.142480+0000","last_undegraded":"2024-09-21T05:45:39.142480+0000","last_fullsized":"2024-09-21T05:45:39.142480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255021+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:45:14.255021+0000","last_peered":"2024-09-21T05:45:14.255021+0000","last_clean":"2024-09-21T05:45:14.255021+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:45:14.255021+0000","last_undegraded":"2024-09-21T05:45:14.255021+0000","last_fullsized":"2024-09-21T05:45:14.255021+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255492+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:45:14.255492+0000","last_peered":"2024-09-21T05:45:14.255492+0000","last_clean":"2024-09-21T05:45:14.255492+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:45:14.255492+0000","last_undegraded":"2024-09-21T05:45:14.255492+0000","last_fullsized":"2024-09-21T05:45:14.255492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"71'1","reported_seq":19,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254977+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:45:14.254977+0000","last_peered":"2024-09-21T05:45:14.254977+0000","last_clean":"2024-09-21T05:45:14.254977+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:45:14.254977+0000","last_undegraded":"2024-09-21T05:45:14.254977+0000","last_fullsized":"2024-09-21T05:45:14.254977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.269448+0000","last_change":"2024-09-21T05:45:14.268855+0000","last_active":"2024-09-21T05:45:14.269448+0000","last_peered":"2024-09-21T05:45:14.269448+0000","last_clean":"2024-09-21T05:45:14.269448+0000","last_became_active":"2024-09-21T05:45:14.261558+0000","last_became_peered":"2024-09-21T05:45:14.261558+0000","last_unstale":"2024-09-21T05:45:14.269448+0000","last_undegraded":"2024-09-21T05:45:14.269448+0000","last_fullsized":"2024-09-21T05:45:14.269448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:57.978428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252449+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:45:14.252449+0000","last_peered":"2024-09-21T05:45:14.252449+0000","last_clean":"2024-09-21T05:45:14.252449+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:45:14.252449+0000","last_undegraded":"2024-09-21T05:45:14.252449+0000","last_fullsized":"2024-09-21T05:45:14.252449+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"71'14","reported_seq":37,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255620+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:14.255620+0000","last_peered":"2024-09-21T05:45:14.255620+0000","last_clean":"2024-09-21T05:45:14.255620+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:14.255620+0000","last_undegraded":"2024-09-21T05:45:14.255620+0000","last_fullsized":"2024-09-21T05:45:14.255620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.261566+0000","last_change":"2024-09-21T05:45:14.261447+0000","last_active":"2024-09-21T05:45:14.261566+0000","last_peered":"2024-09-21T05:45:14.261566+0000","last_clean":"2024-09-21T05:45:14.261566+0000","last_became_active":"2024-09-21T05:45:14.261072+0000","last_became_peered":"2024-09-21T05:45:14.261072+0000","last_unstale":"2024-09-21T05:45:14.261566+0000","last_undegraded":"2024-09-21T05:45:14.261566+0000","last_fullsized":"2024-09-21T05:45:14.261566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:46.148923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254328+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:45:14.254328+0000","last_peered":"2024-09-21T05:45:14.254328+0000","last_clean":"2024-09-21T05:45:14.254328+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:45:14.254328+0000","last_undegraded":"2024-09-21T05:45:14.254328+0000","last_fullsized":"2024-09-21T05:45:14.254328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260424+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:45:14.260424+0000","last_peered":"2024-09-21T05:45:14.260424+0000","last_clean":"2024-09-21T05:45:14.260424+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:45:14.260424+0000","last_undegraded":"2024-09-21T05:45:14.260424+0000","last_fullsized":"2024-09-21T05:45:14.260424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262195+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:45:14.262195+0000","last_peered":"2024-09-21T05:45:14.262195+0000","last_clean":"2024-09-21T05:45:14.262195+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:45:14.262195+0000","last_undegraded":"2024-09-21T05:45:14.262195+0000","last_fullsized":"2024-09-21T05:45:14.262195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258557+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:45:14.258557+0000","last_peered":"2024-09-21T05:45:14.258557+0000","last_clean":"2024-09-21T05:45:14.258557+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:45:14.258557+0000","last_undegraded":"2024-09-21T05:45:14.258557+0000","last_fullsized":"2024-09-21T05:45:14.258557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321694+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:45:14.321694+0000","last_peered":"2024-09-21T05:45:14.321694+0000","last_clean":"2024-09-21T05:45:14.321694+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:45:14.321694+0000","last_undegraded":"2024-09-21T05:45:14.321694+0000","last_fullsized":"2024-09-21T05:45:14.321694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256737+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:45:14.256737+0000","last_peered":"2024-09-21T05:45:14.256737+0000","last_clean":"2024-09-21T05:45:14.256737+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:45:14.256737+0000","last_undegraded":"2024-09-21T05:45:14.256737+0000","last_fullsized":"2024-09-21T05:45:14.256737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256606+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:45:14.256606+0000","last_peered":"2024-09-21T05:45:14.256606+0000","last_clean":"2024-09-21T05:45:14.256606+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:45:14.256606+0000","last_undegraded":"2024-09-21T05:45:14.256606+0000","last_fullsized":"2024-09-21T05:45:14.256606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321933+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:45:14.321933+0000","last_peered":"2024-09-21T05:45:14.321933+0000","last_clean":"2024-09-21T05:45:14.321933+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:45:14.321933+0000","last_undegraded":"2024-09-21T05:45:14.321933+0000","last_fullsized":"2024-09-21T05:45:14.321933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258860+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:45:14.258860+0000","last_peered":"2024-09-21T05:45:14.258860+0000","last_clean":"2024-09-21T05:45:14.258860+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:45:14.258860+0000","last_undegraded":"2024-09-21T05:45:14.258860+0000","last_fullsized":"2024-09-21T05:45:14.258860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256438+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:45:14.256438+0000","last_peered":"2024-09-21T05:45:14.256438+0000","last_clean":"2024-09-21T05:45:14.256438+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:45:14.256438+0000","last_undegraded":"2024-09-21T05:45:14.256438+0000","last_fullsized":"2024-09-21T05:45:14.256438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.263778+0000","last_change":"2024-09-21T05:45:14.263694+0000","last_active":"2024-09-21T05:45:14.263778+0000","last_peered":"2024-09-21T05:45:14.263778+0000","last_clean":"2024-09-21T05:45:14.263778+0000","last_became_active":"2024-09-21T05:45:14.263394+0000","last_became_peered":"2024-09-21T05:45:14.263394+0000","last_unstale":"2024-09-21T05:45:14.263778+0000","last_undegraded":"2024-09-21T05:45:14.263778+0000","last_fullsized":"2024-09-21T05:45:14.263778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:59.214553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260360+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:45:14.260360+0000","last_peered":"2024-09-21T05:45:14.260360+0000","last_clean":"2024-09-21T05:45:14.260360+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:45:14.260360+0000","last_undegraded":"2024-09-21T05:45:14.260360+0000","last_fullsized":"2024-09-21T05:45:14.260360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262259+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:45:14.262259+0000","last_peered":"2024-09-21T05:45:14.262259+0000","last_clean":"2024-09-21T05:45:14.262259+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:45:14.262259+0000","last_undegraded":"2024-09-21T05:45:14.262259+0000","last_fullsized":"2024-09-21T05:45:14.262259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":36,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258852+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:45:14.258852+0000","last_peered":"2024-09-21T05:45:14.258852+0000","last_clean":"2024-09-21T05:45:14.258852+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:45:14.258852+0000","last_undegraded":"2024-09-21T05:45:14.258852+0000","last_fullsized":"2024-09-21T05:45:14.258852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260516+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:45:14.260516+0000","last_peered":"2024-09-21T05:45:14.260516+0000","last_clean":"2024-09-21T05:45:14.260516+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:45:14.260516+0000","last_undegraded":"2024-09-21T05:45:14.260516+0000","last_fullsized":"2024-09-21T05:45:14.260516+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322148+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:45:14.322148+0000","last_peered":"2024-09-21T05:45:14.322148+0000","last_clean":"2024-09-21T05:45:14.322148+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:45:14.322148+0000","last_undegraded":"2024-09-21T05:45:14.322148+0000","last_fullsized":"2024-09-21T05:45:14.322148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322040+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:45:14.322040+0000","last_peered":"2024-09-21T05:45:14.322040+0000","last_clean":"2024-09-21T05:45:14.322040+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:45:14.322040+0000","last_undegraded":"2024-09-21T05:45:14.322040+0000","last_fullsized":"2024-09-21T05:45:14.322040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:42.146099+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:45:42.146099+0000","last_peered":"2024-09-21T05:45:42.146099+0000","last_clean":"2024-09-21T05:45:42.146099+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:42.146099+0000","last_undegraded":"2024-09-21T05:45:42.146099+0000","last_fullsized":"2024-09-21T05:45:42.146099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71570,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":533,"num_read_kb":350,"num_write":241,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1316,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874267,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":819200,"data_stored":399975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037791,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32668,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712484,"statfs":{"total":95995035648,"available":95961583616,"internally_reserved":0,"allocated":1511424,"data_stored":1054676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201316,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36164,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708988,"statfs":{"total":95995035648,"available":95958003712,"internally_reserved":0,"allocated":831488,"data_stored":398820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":819200,"data_stored":399086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36744,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708408,"statfs":{"total":95995035648,"available":95957409792,"internally_reserved":0,"allocated":1425408,"data_stored":991325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724594,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":815104,"data_stored":398537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888119,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1433600,"data_stored":990421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":823296,"data_stored":399020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215168,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":831488,"data_stored":398362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":892928,"data_stored":464196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":880640,"data_stored":464207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":835584,"data_stored":400267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:45:46.498 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T05:45:46.498 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T05:45:46.498 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T05:45:46.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-21T05:45:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:46 smithi129 ceph-mon[34885]: pgmap v287: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:46.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:46 smithi181 ceph-mon[34819]: pgmap v287: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:47.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 -- 172.21.15.28:0/1957958111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 msgr2=0x7feac8104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 --2- 172.21.15.28:0/1957958111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac8104dc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 -- 172.21.15.28:0/1957958111 shutdown_connections 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 --2- 172.21.15.28:0/1957958111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 0x7feac8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 --2- 172.21.15.28:0/1957958111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 0x7feac810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 --2- 172.21.15.28:0/1957958111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac8104dc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 -- 172.21.15.28:0/1957958111 >> 172.21.15.28:0/1957958111 conn(0x7feac8079730 msgr2=0x7feac8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 -- 172.21.15.28:0/1957958111 shutdown_connections 2024-09-21T05:45:47.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.200+0000 7fead074f640 1 -- 172.21.15.28:0/1957958111 wait complete. 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.201+0000 7fead074f640 1 Processor -- start 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.201+0000 7fead074f640 1 -- start start 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 0x7feac819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 0x7feac819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feac8113a70 con 0x7feac81049b0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feac81138f0 con 0x7feac8105300 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7fead074f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feac8113bf0 con 0x7feac810de10 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53758/0 (socket says 172.21.15.28:53758) 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 -- 172.21.15.28:0/85293046 learned_addr learned my addr 172.21.15.28:0/85293046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feacecec640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 0x7feac819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feacdcea640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 0x7feac819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 -- 172.21.15.28:0/85293046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 msgr2=0x7feac819fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 0x7feac819fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 -- 172.21.15.28:0/85293046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 msgr2=0x7feac819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 0x7feac819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 -- 172.21.15.28:0/85293046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac81a0410 con 0x7feac81049b0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.202+0000 7feace4eb640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feac0032fd0 tx=0x7feac0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.203+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac003f030 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.203+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac81a06a0 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.203+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feac0004490 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.203+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0035070 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.203+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac81a6e60 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.204+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac806b560 con 0x7feac81049b0 2024-09-21T05:45:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.205+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7feac0042030 con 0x7feac81049b0 2024-09-21T05:45:47.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.206+0000 7feac77fe640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 0x7feab00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:47.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.206+0000 7feacdcea640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 0x7feab00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:47.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.207+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7feac0107bd0 con 0x7feac81049b0 2024-09-21T05:45:47.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.207+0000 7feacdcea640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 0x7feab00b8b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.211+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feac00ce1a0 con 0x7feac81049b0 2024-09-21T05:45:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.441+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7feac8108f40 con 0x7feac81049b0 2024-09-21T05:45:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.441+0000 7feac77fe640 1 -- 172.21.15.28:0/85293046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7feac00d2580 con 0x7feac81049b0 2024-09-21T05:45:47.443 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:47.443 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T05:45:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 msgr2=0x7feab00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 0x7feab00b8b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc005850 comp rx=0 tx=0).stop 2024-09-21T05:45:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 msgr2=0x7feac819a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feac0032fd0 tx=0x7feac0004270 comp rx=0 tx=0).stop 2024-09-21T05:45:47.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:47 smithi028 ceph-mon[31227]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 shutdown_connections 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feac810de10 0x7feac819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feac8105300 0x7feac819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7feab00b65f0 0x7feab00b8b00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 --2- 172.21.15.28:0/85293046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac81049b0 0x7feac819a4a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 >> 172.21.15.28:0/85293046 conn(0x7feac8079730 msgr2=0x7feac8100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 shutdown_connections 2024-09-21T05:45:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:47.444+0000 7fead074f640 1 -- 172.21.15.28:0/85293046 wait complete. 2024-09-21T05:45:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:47 smithi129 ceph-mon[34885]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:47.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:47 smithi181 ceph-mon[34819]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:45:48.072 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T05:45:48.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:45:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:48 smithi129 ceph-mon[34885]: pgmap v288: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:48 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/85293046' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T05:45:48.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:48 smithi028 ceph-mon[31227]: pgmap v288: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:48.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:48 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/85293046' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T05:45:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 -- 172.21.15.28:0/653635418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c075dd0 msgr2=0x7fe10c0761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 --2- 172.21.15.28:0/653635418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c075dd0 0x7fe10c0761e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe100009900 tx=0x7fe10002a410 comp rx=0 tx=0).stop 2024-09-21T05:45:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 -- 172.21.15.28:0/653635418 shutdown_connections 2024-09-21T05:45:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 --2- 172.21.15.28:0/653635418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c114430 0x7fe10c116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 --2- 172.21.15.28:0/653635418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c075dd0 0x7fe10c0761e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 --2- 172.21.15.28:0/653635418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 0x7fe10c0757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 -- 172.21.15.28:0/653635418 >> 172.21.15.28:0/653635418 conn(0x7fe10c07b850 msgr2=0x7fe10c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 -- 172.21.15.28:0/653635418 shutdown_connections 2024-09-21T05:45:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.757+0000 7fe1136da640 1 -- 172.21.15.28:0/653635418 wait complete. 2024-09-21T05:45:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 Processor -- start 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 -- start start 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c075dd0 0x7fe10c1a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 0x7fe10c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c11a040 con 0x7fe10c114430 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c119ec0 con 0x7fe10c075dd0 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe1136da640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c11a1c0 con 0x7fe10c077380 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe111c77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe111c77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53776/0 (socket says 172.21.15.28:53776) 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe111c77640 1 -- 172.21.15.28:0/176830866 learned_addr learned my addr 172.21.15.28:0/176830866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe111476640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c075dd0 0x7fe10c1a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.758+0000 7fe110c75640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 0x7fe10c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 -- 172.21.15.28:0/176830866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 msgr2=0x7fe10c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 0x7fe10c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 -- 172.21.15.28:0/176830866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c075dd0 msgr2=0x7fe10c1a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c075dd0 0x7fe10c1a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 -- 172.21.15.28:0/176830866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c1a6a40 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe111c77640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe10400eaa0 tx=0x7fe10400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10400cdb0 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10c1a6d30 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe104005020 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040116d0 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.759+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10c1ad490 con 0x7fe10c114430 2024-09-21T05:45:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.760+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10c076cc0 con 0x7fe10c114430 2024-09-21T05:45:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.762+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fe1040051d0 con 0x7fe10c114430 2024-09-21T05:45:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.763+0000 7fe10a7fc640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 0x7fe0f40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.763+0000 7fe111476640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 0x7fe0f40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.763+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe1040e7b20 con 0x7fe10c114430 2024-09-21T05:45:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.763+0000 7fe111476640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 0x7fe0f40b8b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0058c0 tx=0x7fe0fc00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.766+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe104016030 con 0x7fe10c114430 2024-09-21T05:45:48.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:48 smithi181 ceph-mon[34819]: pgmap v288: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:48.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:48 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/85293046' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T05:45:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.991+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe10c0761e0 con 0x7fe10c114430 2024-09-21T05:45:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.992+0000 7fe10a7fc640 1 -- 172.21.15.28:0/176830866 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7fe1040b24d0 con 0x7fe10c114430 2024-09-21T05:45:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:48.994 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:45:48.994 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 msgr2=0x7fe0f40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 0x7fe0f40b8b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0058c0 tx=0x7fe0fc00a2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 msgr2=0x7fe10c1a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe10400eaa0 tx=0x7fe10400ef60 comp rx=0 tx=0).stop 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 shutdown_connections 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe0f40b65f0 0x7fe0f40b8b00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10c114430 0x7fe10c1a6310 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe10c077380 0x7fe10c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 --2- 172.21.15.28:0/176830866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe10c075dd0 0x7fe10c1a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 >> 172.21.15.28:0/176830866 conn(0x7fe10c07b850 msgr2=0x7fe10c113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 shutdown_connections 2024-09-21T05:45:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:48.994+0000 7fe1136da640 1 -- 172.21.15.28:0/176830866 wait complete. 2024-09-21T05:45:49.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:49 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/176830866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:49.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:45:49.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:49 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/176830866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:49.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:49 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/176830866' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 -- 172.21.15.28:0/555145758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8678109e10 msgr2=0x7f8678110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 --2- 172.21.15.28:0/555145758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8678109e10 0x7f8678110640 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8670009ca0 tx=0x7f867002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 -- 172.21.15.28:0/555145758 shutdown_connections 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 --2- 172.21.15.28:0/555145758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8678109e10 0x7f8678110640 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 --2- 172.21.15.28:0/555145758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86781094c0 0x7f86781098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 --2- 172.21.15.28:0/555145758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678103b80 0x7f8678103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 -- 172.21.15.28:0/555145758 >> 172.21.15.28:0/555145758 conn(0x7f86780ff610 msgr2=0x7f8678101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 -- 172.21.15.28:0/555145758 shutdown_connections 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.299+0000 7f867ccdc640 1 -- 172.21.15.28:0/555145758 wait complete. 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 Processor -- start 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 -- start start 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 0x7f867819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 0x7f867819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8678113b20 con 0x7f86781094c0 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86781139a0 con 0x7f8678103b80 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867ccdc640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8678113ca0 con 0x7f8678109e10 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53792/0 (socket says 172.21.15.28:53792) 2024-09-21T05:45:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 -- 172.21.15.28:0/1632777849 learned_addr learned my addr 172.21.15.28:0/1632777849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8676d9d640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 0x7f867819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867659c640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 0x7f867819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 -- 172.21.15.28:0/1632777849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 msgr2=0x7f867819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 0x7f867819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 -- 172.21.15.28:0/1632777849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 msgr2=0x7f867819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 0x7f867819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8675d9b640 1 -- 172.21.15.28:0/1632777849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86781a03c0 con 0x7f86781094c0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f8676d9d640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 0x7f867819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.301+0000 7f867659c640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 0x7f867819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f8675d9b640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f866c00d740 tx=0x7f866c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c017070 con 0x7f86781094c0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866c00de50 con 0x7f86781094c0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86781a06b0 con 0x7f86781094c0 2024-09-21T05:45:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c00f620 con 0x7f86781094c0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.302+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86781a6e10 con 0x7f86781094c0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.303+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8678103f90 con 0x7f86781094c0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.305+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f866c01a070 con 0x7f86781094c0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.305+0000 7f86677fe640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 0x7f86600b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.305+0000 7f867659c640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 0x7f86600b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.306+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f866c0e7080 con 0x7f86781094c0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.306+0000 7f867659c640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 0x7f86600b8b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8668009b40 tx=0x7f8668005e50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.308+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f866c0ad6b0 con 0x7f86781094c0 2024-09-21T05:45:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.534+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f867802e640 con 0x7f86781094c0 2024-09-21T05:45:50.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.535+0000 7f86677fe640 1 -- 172.21.15.28:0/1632777849 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7f866c0b1a90 con 0x7f86781094c0 2024-09-21T05:45:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:50.537 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:45:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:45:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 msgr2=0x7f86600b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 0x7f86600b8b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8668009b40 tx=0x7f8668005e50 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 msgr2=0x7f867819a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f866c00d740 tx=0x7f866c00dc00 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 shutdown_connections 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8678109e10 0x7f867819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86600b65f0 0x7f86600b8b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86781094c0 0x7f867819a9f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 --2- 172.21.15.28:0/1632777849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8678103b80 0x7f867819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.537+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 >> 172.21.15.28:0/1632777849 conn(0x7f86780ff610 msgr2=0x7f8678108ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.538+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 shutdown_connections 2024-09-21T05:45:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:50.538+0000 7f867ccdc640 1 -- 172.21.15.28:0/1632777849 wait complete. 2024-09-21T05:45:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:50 smithi028 ceph-mon[31227]: pgmap v289: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:50 smithi181 ceph-mon[34819]: pgmap v289: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:50 smithi129 ceph-mon[34885]: pgmap v289: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:45:51.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:45:51.519 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:51.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:51 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1632777849' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:51.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:51 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1632777849' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:51.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:51 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1632777849' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/2023418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 msgr2=0x7fa0bc108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/2023418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc108800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009900 tx=0x7fa0ac02a410 comp rx=0 tx=0).stop 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/2023418052 shutdown_connections 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/2023418052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc108d40 0x7fa0bc10f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/2023418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc108800 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/2023418052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc102ab0 0x7fa0bc102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/2023418052 >> 172.21.15.28:0/2023418052 conn(0x7fa0bc0797c0 msgr2=0x7fa0bc079bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/2023418052 shutdown_connections 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.019+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/2023418052 wait complete. 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 Processor -- start 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 -- start start 2024-09-21T05:45:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc102ab0 0x7fa0bc19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 0x7fa0bc19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc112a00 con 0x7fa0bc1083f0 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc112880 con 0x7fa0bc108d40 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c2fb4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc112b80 con 0x7fa0bc102ab0 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53812/0 (socket says 172.21.15.28:53812) 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0b3fff640 1 -- 172.21.15.28:0/3924460314 learned_addr learned my addr 172.21.15.28:0/3924460314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.020+0000 7fa0c0d50640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc102ab0 0x7fa0bc19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0c1551640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 0x7fa0bc19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 -- 172.21.15.28:0/3924460314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc102ab0 msgr2=0x7fa0bc19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc102ab0 0x7fa0bc19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 -- 172.21.15.28:0/3924460314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 msgr2=0x7fa0bc19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 0x7fa0bc19bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 -- 172.21.15.28:0/3924460314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc1a64f0 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0c1551640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 0x7fa0bc19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b3fff640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac02a930 tx=0x7fa0ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac03f030 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc1a6780 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0ac003cb0 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac035070 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.021+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc1a6cb0 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.023+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0bc103d30 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.024+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fa0ac042030 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.025+0000 7fa0b1ffb640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 0x7fa0a40b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.025+0000 7fa0c0d50640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 0x7fa0a40b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.025+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa0ac108050 con 0x7fa0bc1083f0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.025+0000 7fa0c0d50640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 0x7fa0a40b8b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80058c0 tx=0x7fa0b800a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:52.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.028+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa0ac0ce680 con 0x7fa0bc1083f0 2024-09-21T05:45:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.219+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0bc102ec0 con 0x7fa0bc1083f0 2024-09-21T05:45:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.221+0000 7fa0b1ffb640 1 -- 172.21.15.28:0/3924460314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+23122 (secure 0 0 0) 0x7fa0bc102ec0 con 0x7fa0bc1083f0 2024-09-21T05:45:52.223 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:52.223 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:14.248651+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 msgr2=0x7fa0a40b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 0x7fa0a40b8b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80058c0 tx=0x7fa0b800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 msgr2=0x7fa0bc19b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac02a930 tx=0x7fa0ac004270 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 shutdown_connections 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0bc108d40 0x7fa0bc19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa0a40b6620 0x7fa0a40b8b30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc1083f0 0x7fa0bc19b830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 --2- 172.21.15.28:0/3924460314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0bc102ab0 0x7fa0bc19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.223+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 >> 172.21.15.28:0/3924460314 conn(0x7fa0bc0797c0 msgr2=0x7fa0bc104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.224+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 shutdown_connections 2024-09-21T05:45:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:52.224+0000 7fa0c2fb4640 1 -- 172.21.15.28:0/3924460314 wait complete. 2024-09-21T05:45:52.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:52 smithi028 ceph-mon[31227]: pgmap v290: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:52.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:52 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3924460314' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:52 smithi181 ceph-mon[34819]: pgmap v290: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:52 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3924460314' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:52 smithi129 ceph-mon[34885]: pgmap v290: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:45:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:52 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3924460314' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:45:52.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:45:53.161 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 -- 172.21.15.28:0/1570619419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 msgr2=0x7f28ac10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 --2- 172.21.15.28:0/1570619419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac10f520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009ca0 tx=0x7f28a402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 -- 172.21.15.28:0/1570619419 shutdown_connections 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 --2- 172.21.15.28:0/1570619419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac10f520 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 --2- 172.21.15.28:0/1570619419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 --2- 172.21.15.28:0/1570619419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 -- 172.21.15.28:0/1570619419 >> 172.21.15.28:0/1570619419 conn(0x7f28ac079730 msgr2=0x7f28ac079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 -- 172.21.15.28:0/1570619419 shutdown_connections 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.691+0000 7f28b1010640 1 -- 172.21.15.28:0/1570619419 wait complete. 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.692+0000 7f28b1010640 1 Processor -- start 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.692+0000 7f28b1010640 1 -- start start 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.692+0000 7f28b1010640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.692+0000 7f28b1010640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28b1010640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28b1010640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac113a80 con 0x7f28ac108cf0 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28b1010640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac113900 con 0x7f28ac102a60 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28b1010640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac113c00 con 0x7f28ac1083a0 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53834/0 (socket says 172.21.15.28:53834) 2024-09-21T05:45:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 -- 172.21.15.28:0/1481619536 learned_addr learned my addr 172.21.15.28:0/1481619536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28aa59c640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28aad9d640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 -- 172.21.15.28:0/1481619536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 msgr2=0x7f28ac19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 -- 172.21.15.28:0/1481619536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 msgr2=0x7f28ac19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 -- 172.21.15.28:0/1481619536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac1a0380 con 0x7f28ac108cf0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28aa59c640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28aad9d640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f28ab59e640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005cc0 tx=0x7f28a402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.693+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4003bf0 con 0x7f28ac108cf0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.694+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac1a0610 con 0x7f28ac108cf0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.694+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a4003d90 con 0x7f28ac108cf0 2024-09-21T05:45:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.694+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a402fbc0 con 0x7f28ac108cf0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.694+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac1a6dd0 con 0x7f28ac108cf0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.695+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac102e70 con 0x7f28ac108cf0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.696+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f28a402fd60 con 0x7f28ac108cf0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.697+0000 7f2893fff640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 0x7f28940b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.697+0000 7f28aad9d640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 0x7f28940b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.697+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f28a4108bf0 con 0x7f28ac108cf0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.697+0000 7f28aad9d640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 0x7f28940b8b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f289c005d00 tx=0x7f289c005c70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.701+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f28a40cf1c0 con 0x7f28ac108cf0 2024-09-21T05:45:53.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.925+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28ac10aae0 con 0x7f28ac108cf0 2024-09-21T05:45:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.926+0000 7f2893fff640 1 -- 172.21.15.28:0/1481619536 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7f28a40098b0 con 0x7f28ac108cf0 2024-09-21T05:45:53.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:45:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:45:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:45:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 msgr2=0x7f28940b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 0x7f28940b8b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f289c005d00 tx=0x7f289c005c70 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 msgr2=0x7f28ac19fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005cc0 tx=0x7f28a402f380 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 shutdown_connections 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f28940b65f0 0x7f28940b8b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac108cf0 0x7f28ac19fc50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28ac1083a0 0x7f28ac19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 --2- 172.21.15.28:0/1481619536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28ac102a60 0x7f28ac19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 >> 172.21.15.28:0/1481619536 conn(0x7f28ac079730 msgr2=0x7f28ac105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 shutdown_connections 2024-09-21T05:45:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:53.929+0000 7f28b1010640 1 -- 172.21.15.28:0/1481619536 wait complete. 2024-09-21T05:45:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:54 smithi028 ceph-mon[31227]: pgmap v291: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:54 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1481619536' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:54.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T05:44:39.761973+0000', 'modified': '2024-09-21T05:44:52.047653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24688, 'mds_1': 24680, 'mds_2': 24673, 'mds_3': 24664, 'mds_4': 24653}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14850': {'gid': 14850, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/2924711217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2924711217}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2924711217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14856': {'gid': 14856, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2765835727', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2765835727}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2765835727}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24614': {'gid': 24614, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6833/2193748712', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2193748712}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 2193748712}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24622': {'gid': 24622, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6833/1938433457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 1938433457}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 1938433457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24638': {'gid': 24638, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6835/2842004697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 2842004697}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2842004697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24653': {'gid': 24653, 'name': 'h', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.129:6837/2283221342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 2283221342}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 2283221342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24664': {'gid': 24664, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.181:6835/1316004847', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 1316004847}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 1316004847}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24673': {'gid': 24673, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.181:6837/2388771333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 2388771333}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2388771333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24680': {'gid': 24680, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6839/3631196486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 3631196486}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 3631196486}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24688': {'gid': 24688, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.181:6839/3430071220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3430071220}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3430071220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24688, 'qdb_cluster': [24653, 24664, 24673, 24680, 24688]} 2024-09-21T05:45:54.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T05:45:54.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T05:45:54.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:54 smithi129 ceph-mon[34885]: pgmap v291: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:54.667 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:54 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1481619536' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:54 smithi181 ceph-mon[34819]: pgmap v291: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:45:54.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:54 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1481619536' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:45:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 -- 172.21.15.28:0/2425709498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe8108d20 msgr2=0x7fcbe810f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2425709498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe8108d20 0x7fcbe810f550 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009ca0 tx=0x7fcbe002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 -- 172.21.15.28:0/2425709498 shutdown_connections 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2425709498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe8108d20 0x7fcbe810f550 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2425709498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe81083d0 0x7fcbe81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2425709498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 0x7fcbe8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 -- 172.21.15.28:0/2425709498 >> 172.21.15.28:0/2425709498 conn(0x7fcbe8079760 msgr2=0x7fcbe8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 -- 172.21.15.28:0/2425709498 shutdown_connections 2024-09-21T05:45:55.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 -- 172.21.15.28:0/2425709498 wait complete. 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.365+0000 7fcbf0135640 1 Processor -- start 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 -- start start 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 0x7fcbe8105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe8108d20 0x7fcbe8105b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe81129f0 con 0x7fcbe81083d0 2024-09-21T05:45:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8112870 con 0x7fcbe8102ae0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbf0135640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8112b70 con 0x7fcbe8108d20 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55134/0 (socket says 172.21.15.28:55134) 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 -- 172.21.15.28:0/2645246922 learned_addr learned my addr 172.21.15.28:0/2645246922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe8108d20 0x7fcbe8105b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 0x7fcbe8105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 -- 172.21.15.28:0/2645246922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe8108d20 msgr2=0x7fcbe8105b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe8108d20 0x7fcbe8105b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 -- 172.21.15.28:0/2645246922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 msgr2=0x7fcbe8105110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 0x7fcbe8105110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.366+0000 7fcbed6d0640 1 -- 172.21.15.28:0/2645246922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe81a8740 con 0x7fcbe81083d0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c970 tx=0x7fcbd800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbdeffd640 1 -- 172.21.15.28:0/2645246922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8007bb0 con 0x7fcbe81083d0 2024-09-21T05:45:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe81a8a30 con 0x7fcbe81083d0 2024-09-21T05:45:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbdeffd640 1 -- 172.21.15.28:0/2645246922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd8007d50 con 0x7fcbe81083d0 2024-09-21T05:45:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe81a8fc0 con 0x7fcbe81083d0 2024-09-21T05:45:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.367+0000 7fcbdeffd640 1 -- 172.21.15.28:0/2645246922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd800f450 con 0x7fcbe81083d0 2024-09-21T05:45:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 Processor -- start 2024-09-21T05:45:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 -- start start 2024-09-21T05:45:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbdeffd640 1 -- 172.21.15.28:0/2645246922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fcbd8004ec0 con 0x7fcbe81083d0 2024-09-21T05:45:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d20440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d20980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637b5c97c10 con 0x5637b59510a0 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5637b5c87040 con 0x5637b5959280 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5637b5c8ae50 con 0x5637b5979f60 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbee6d2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbeded1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d20440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbee6d2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44712/0 (socket says 172.21.15.28:44712) 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbee6d2640 1 -- 172.21.15.28:0/938914828 learned_addr learned my addr 172.21.15.28:0/938914828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.370+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d20980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 -- 172.21.15.28:0/938914828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 msgr2=0x5637b5d20980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d20980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 -- 172.21.15.28:0/938914828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 msgr2=0x5637b5d20440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d20440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 -- 172.21.15.28:0/938914828 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b59e5540 con 0x5637b5979f60 2024-09-21T05:45:55.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002f4c0 tx=0x7fcbe0031af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d642c1401b6828 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbdeffd640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 0x7fcbd40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 0x7fcbd40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbd3fff640 1 -- 172.21.15.28:0/938914828 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0031450 con 0x5637b5979f60 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbdeffd640 1 -- 172.21.15.28:0/2645246922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcbd80e7290 con 0x7fcbe81083d0 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbd3fff640 1 -- 172.21.15.28:0/938914828 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe0036030 con 0x5637b5979f60 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.371+0000 7fcbd3fff640 1 -- 172.21.15.28:0/938914828 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0004360 con 0x5637b5979f60 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 -- 172.21.15.28:0/938914828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 msgr2=0x5637b5d1ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 0x7fcbd40b8b80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5637b5d1eb80 tx=0x7fcbe4008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002f4c0 tx=0x7fcbe0031af0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 -- 172.21.15.28:0/938914828 shutdown_connections 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d20980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d20440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/938914828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d1ff00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 -- 172.21.15.28:0/938914828 >> 172.21.15.28:0/938914828 conn(0x5637b596e350 msgr2=0x5637b5d1ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 -- 172.21.15.28:0/938914828 shutdown_connections 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.372+0000 7fcbf66a9740 1 -- 172.21.15.28:0/938914828 wait complete. 2024-09-21T05:45:55.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 -- start start 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 Processor -- start 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d377f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637b5ae85f0 con 0x5637b59510a0 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5637b5c22010 con 0x5637b5959280 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.373+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d2e500 con 0x5637b5979f60 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbf66a9740 10 client.0 _ll_register_callbacks cb 0x5637b5c81340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55158/0 (socket says 172.21.15.28:55158) 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 -- 172.21.15.28:0/328367319 learned_addr learned my addr 172.21.15.28:0/328367319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 -- 172.21.15.28:0/328367319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 msgr2=0x5637b5d377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d377f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 -- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 msgr2=0x5637b5d2bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d2bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 -- 172.21.15.28:0/328367319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b5c4fa00 con 0x5637b59510a0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbeded1640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4016a00 tx=0x7fcbe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d235a53c372e0e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe401c050 con 0x5637b59510a0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcbe4014b40 con 0x5637b59510a0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.374+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbe0036030 con 0x5637b59510a0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbe4010e70 con 0x5637b59510a0 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 10 client.15219 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:55.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5637b5d3a430 con 0x5637b59510a0 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe0036030 con 0x5637b59510a0 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe4006550 con 0x5637b59510a0 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 10 client.15219 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 10 client.15219 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 20 client.15219 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b59388e0 0x5637b5d3df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 4 client.15219 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.375+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637b5c99b60 con 0x5637b59388e0 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.376+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b59388e0 0x5637b5d3df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.376+0000 7fcbd17fa640 10 client.15219 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.376+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b59388e0 0x5637b5d3df10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3122ba9cf988fe15 server_cookie=4aae2c17b6ea187e in_seq=0 out_seq=0 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbd17fa640 1 -- 172.21.15.28:0/328367319 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5637b5c99b60 con 0x5637b59388e0 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbd17fa640 10 client.15219 handle_command_reply: tid=0 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 client.15219 shutdown 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 msgr2=0x5637b5d2b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4016a00 tx=0x7fcbe40046c0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 shutdown_connections 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b59388e0 0x5637b5d3df10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5979f60 0x5637b5d377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbd17fa640 0 client.15219 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:45:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d2bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/328367319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d2b980 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.377+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 >> 172.21.15.28:0/328367319 conn(0x5637b596e350 msgr2=0x5637b5d23090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.378+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 shutdown_connections 2024-09-21T05:45:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.378+0000 7fcbf66a9740 1 -- 172.21.15.28:0/328367319 wait complete. 2024-09-21T05:45:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.378+0000 7fcbf66a9740 20 client.15219 trim_cache size 0 max 16384 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.396+0000 7fcbf66a9740 1 Processor -- start 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.396+0000 7fcbf66a9740 1 -- start start 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 0x5637b5d26e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5959280 0x5637b5d2a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3c250 con 0x5637b59388e0 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3c920 con 0x5637b59510a0 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3cff0 con 0x5637b5959280 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55168/0 (socket says 172.21.15.28:55168) 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 -- 172.21.15.28:0/2251518647 learned_addr learned my addr 172.21.15.28:0/2251518647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 0x5637b5d26e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5959280 0x5637b5d2a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 -- 172.21.15.28:0/2251518647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5959280 msgr2=0x5637b5d2a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5959280 0x5637b5d2a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 -- 172.21.15.28:0/2251518647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 msgr2=0x5637b5d26e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 0x5637b5d26e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbeded1640 1 -- 172.21.15.28:0/2251518647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b5d2ca40 con 0x5637b59388e0 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.397+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 0x5637b5d26e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.398+0000 7fcbeded1640 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4008010 tx=0x7fcbe40166b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df9abe04d20e6fa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.398+0000 7fcbd37fe640 1 -- 172.21.15.28:0/2251518647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4039070 con 0x5637b59388e0 2024-09-21T05:45:55.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.398+0000 7fcbd37fe640 1 -- 172.21.15.28:0/2251518647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe4026070 con 0x5637b59388e0 2024-09-21T05:45:55.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.398+0000 7fcbd37fe640 1 -- 172.21.15.28:0/2251518647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe401c040 con 0x5637b59388e0 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- 172.21.15.28:0/2251518647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 msgr2=0x5637b5d26930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4008010 tx=0x7fcbe40166b0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- 172.21.15.28:0/2251518647 shutdown_connections 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b5959280 0x5637b5d2a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b59510a0 0x5637b5d26e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/2251518647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59388e0 0x5637b5d26930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- 172.21.15.28:0/2251518647 >> 172.21.15.28:0/2251518647 conn(0x5637b596e350 msgr2=0x5637b5d3b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- 172.21.15.28:0/2251518647 shutdown_connections 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- 172.21.15.28:0/2251518647 wait complete. 2024-09-21T05:45:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 -- start start 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.399+0000 7fcbf66a9740 1 Processor -- start 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b59388e0 0x5637b5d46a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d40cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3dd90 con 0x5637b59510a0 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3d6c0 con 0x5637b5959280 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5637b5d3e460 con 0x5637b59388e0 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbf66a9740 10 client.0 _ll_register_callbacks cb 0x7fcbe400b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55178/0 (socket says 172.21.15.28:55178) 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 -- 172.21.15.28:0/297441651 learned_addr learned my addr 172.21.15.28:0/297441651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbeded1640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b59388e0 0x5637b5d46a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbee6d2640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d40cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 -- 172.21.15.28:0/297441651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b59388e0 msgr2=0x5637b5d46a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b59388e0 0x5637b5d46a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 -- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 msgr2=0x5637b5d40cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d40cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 -- 172.21.15.28:0/297441651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b5d41440 con 0x5637b59510a0 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbed6d0640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c940 tx=0x7fcbd80048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b488caa8318f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.400+0000 7fcbd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd801d040 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcbd80a8b10 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbe4026070 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbd80ad8c0 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 10 client.15225 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5637b5da9440 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe4026070 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fcbd80abe90 con 0x5637b59510a0 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 10 client.15225 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 20 client.15225 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:55.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b5979f60 0x5637b5dacce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 4 client.15225 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.401+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "status"}) -- 0x5637b5c95610 con 0x5637b5979f60 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbeded1640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b5979f60 0x5637b5dacce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbeded1640 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b5979f60 0x5637b5dacce0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=25cef487d4f4afbf server_cookie=21ecce8b7c25bcec in_seq=0 out_seq=0 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbd17fa640 10 client.15225 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbd17fa640 1 -- 172.21.15.28:0/297441651 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5637b5c95610 con 0x5637b5979f60 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbd17fa640 10 client.15225 handle_command_reply: tid=0 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbf66a9740 1 client.15225 shutdown 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 msgr2=0x5637b5d406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.402+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c940 tx=0x7fcbd80048a0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 shutdown_connections 2024-09-21T05:45:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5637b5979f60 0x5637b5dacce0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5637b5959280 0x5637b5d40cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637b59510a0 0x5637b5d406f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbd17fa640 0 client.15225 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 --2- 172.21.15.28:0/297441651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637b59388e0 0x5637b5d46a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 >> 172.21.15.28:0/297441651 conn(0x5637b596e350 msgr2=0x5637b5d26930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 shutdown_connections 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 1 -- 172.21.15.28:0/297441651 wait complete. 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.403+0000 7fcbf66a9740 20 client.15225 trim_cache size 0 max 16384 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:45:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24653, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 64.202071627999999, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:45:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 96.780841691000006 2024-09-21T05:45:55.413 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:45:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 msgr2=0x7fcbd40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 0x7fcbd40b8b80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5637b5d1eb80 tx=0x7fcbe4008040 comp rx=0 tx=0).stop 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 msgr2=0x7fcbe8105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c970 tx=0x7fcbd800ce30 comp rx=0 tx=0).stop 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 shutdown_connections 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbe8108d20 0x7fcbe8105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcbd40b6670 0x7fcbd40b8b80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbe81083d0 0x7fcbe8105650 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 --2- 172.21.15.28:0/2645246922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbe8102ae0 0x7fcbe8105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 >> 172.21.15.28:0/2645246922 conn(0x7fcbe8079760 msgr2=0x7fcbe806be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 shutdown_connections 2024-09-21T05:45:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:55.404+0000 7fcbf0135640 1 -- 172.21.15.28:0/2645246922 wait complete. 2024-09-21T05:45:56.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.f status 2024-09-21T05:45:56.401 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:56 smithi028 ceph-mon[31227]: pgmap v292: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:56 smithi181 ceph-mon[34819]: pgmap v292: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 -- 172.21.15.28:0/3392984965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888110200 msgr2=0x7f0888112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3392984965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888110200 0x7f0888112620 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0880009ca0 tx=0x7f088002a950 comp rx=0 tx=0).stop 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 -- 172.21.15.28:0/3392984965 shutdown_connections 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3392984965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888110200 0x7f0888112620 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3392984965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102fa0 0x7f088810fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3392984965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888102650 0x7f0888102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 -- 172.21.15.28:0/3392984965 >> 172.21.15.28:0/3392984965 conn(0x7f0888079760 msgr2=0x7f0888079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 -- 172.21.15.28:0/3392984965 shutdown_connections 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.848+0000 7f088fd1a640 1 -- 172.21.15.28:0/3392984965 wait complete. 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 Processor -- start 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 -- start start 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 0x7f088819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888110200 0x7f08881a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0888115ca0 con 0x7f0888102fa0 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0888115b20 con 0x7f0888102650 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088fd1a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0888115e20 con 0x7f0888110200 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55190/0 (socket says 172.21.15.28:55190) 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 -- 172.21.15.28:0/3418887140 learned_addr learned my addr 172.21.15.28:0/3418887140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088dab6640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 0x7f088819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088e2b7640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888110200 0x7f08881a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 -- 172.21.15.28:0/3418887140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888110200 msgr2=0x7f08881a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888110200 0x7f08881a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 -- 172.21.15.28:0/3418887140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 msgr2=0x7f088819c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.849+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 0x7f088819c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f088d2b5640 1 -- 172.21.15.28:0/3418887140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08881a24c0 con 0x7f0888102fa0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f088dab6640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 0x7f088819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f087800c920 tx=0x7f087800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f087effd640 1 -- 172.21.15.28:0/3418887140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878007c70 con 0x7f0888102fa0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f087effd640 1 -- 172.21.15.28:0/3418887140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0878007e10 con 0x7f0888102fa0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08881a27b0 con 0x7f0888102fa0 2024-09-21T05:45:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f087effd640 1 -- 172.21.15.28:0/3418887140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878018820 con 0x7f0888102fa0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.850+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08881a8fd0 con 0x7f0888102fa0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 Processor -- start 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 -- start start 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f087effd640 1 -- 172.21.15.28:0/3418887140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f087800f450 con 0x7f0888102fa0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef50099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef5009a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef50011c10 con 0x55ef4fcd3280 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef50001040 con 0x55ef4fccb0a0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f089628e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ef50004e50 con 0x55ef4fcf3f60 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55206/0 (socket says 172.21.15.28:55206) 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.853+0000 7f088d2b5640 1 -- 172.21.15.28:0/2342669694 learned_addr learned my addr 172.21.15.28:0/2342669694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088e2b7640 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef50099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088dab6640 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef5009a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 -- 172.21.15.28:0/2342669694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 msgr2=0x55ef50099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef50099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 -- 172.21.15.28:0/2342669694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 msgr2=0x55ef5009a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef5009a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 -- 172.21.15.28:0/2342669694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4fd5f540 con 0x55ef4fcd3280 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088d2b5640 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f08780e4f70 tx=0x7f08780edcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd181e86a0ce94a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f087effd640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 0x7f08740b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f0873fff640 1 -- 172.21.15.28:0/2342669694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08780f6070 con 0x55ef4fcd3280 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f088e2b7640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 0x7f08740b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f0873fff640 1 -- 172.21.15.28:0/2342669694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08780ad1c0 con 0x55ef4fcd3280 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f0873fff640 1 -- 172.21.15.28:0/2342669694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878094710 con 0x55ef4fcd3280 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.854+0000 7f087effd640 1 -- 172.21.15.28:0/3418887140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f08780e7780 con 0x7f0888102fa0 2024-09-21T05:45:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 -- 172.21.15.28:0/2342669694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 msgr2=0x55ef5009a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f088e2b7640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 0x7f08740b8bb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0880005cc0 tx=0x7f0880005c30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f08780e4f70 tx=0x7f08780edcb0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 -- 172.21.15.28:0/2342669694 shutdown_connections 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcd3280 0x55ef5009a980 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef5009a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 --2- 172.21.15.28:0/2342669694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef50099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.855+0000 7f089628e740 1 -- 172.21.15.28:0/2342669694 >> 172.21.15.28:0/2342669694 conn(0x55ef4fce8350 msgr2=0x55ef50097df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.856+0000 7f089628e740 1 -- 172.21.15.28:0/2342669694 shutdown_connections 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.856+0000 7f089628e740 1 -- 172.21.15.28:0/2342669694 wait complete. 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.856+0000 7f089628e740 1 -- start start 2024-09-21T05:45:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.856+0000 7f089628e740 1 Processor -- start 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef500b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef4fe625f0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef4ff9c010 con 0x55ef4fcd3280 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ef500a86e0 con 0x55ef4fcf3f60 2024-09-21T05:45:56.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f089628e740 10 client.0 _ll_register_callbacks cb 0x55ef4fffb340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55212/0 (socket says 172.21.15.28:55212) 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 -- 172.21.15.28:0/948073708 learned_addr learned my addr 172.21.15.28:0/948073708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088d2b5640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088e2b7640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef500b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 -- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 msgr2=0x55ef500b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef500b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 -- 172.21.15.28:0/948073708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 msgr2=0x55ef500a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 -- 172.21.15.28:0/948073708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4ffc9a00 con 0x55ef4fccb0a0 2024-09-21T05:45:56.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f088dab6640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0884013790 tx=0x7f0884013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4702250c1f9528e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.857+0000 7f08717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884020060 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0884011dd0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08780ad1c0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f088400f450 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 10 client.15237 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef500b4640 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08780ad1c0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f088401a030 con 0x55ef4fccb0a0 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 10 client.15237 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 10 client.15237 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 20 client.15237 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcb28e0 0x55ef500b8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 4 client.15237 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:45:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.858+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef50013b60 con 0x55ef4fcb28e0 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.859+0000 7f088d2b5640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcb28e0 0x55ef500b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.859+0000 7f088d2b5640 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcb28e0 0x55ef500b8120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=debb8cd85b4cf50a server_cookie=f10e801199bdc2f5 in_seq=0 out_seq=0 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.859+0000 7f08717fa640 10 client.15237 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f08717fa640 1 -- 172.21.15.28:0/948073708 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ef50013b60 con 0x55ef4fcb28e0 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f08717fa640 10 client.15237 handle_command_reply: tid=0 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 client.15237 shutdown 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 msgr2=0x55ef500a59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0884013790 tx=0x7f0884013c50 comp rx=0 tx=0).stop 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 shutdown_connections 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcb28e0 0x55ef500b8120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcf3f60 0x55ef500b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f08717fa640 0 client.15237 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 --2- 172.21.15.28:0/948073708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500a59b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 >> 172.21.15.28:0/948073708 conn(0x55ef4fce8350 msgr2=0x55ef5009d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.860+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 shutdown_connections 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.861+0000 7f089628e740 1 -- 172.21.15.28:0/948073708 wait complete. 2024-09-21T05:45:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.861+0000 7f089628e740 20 client.15237 trim_cache size 0 max 16384 2024-09-21T05:45:56.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:56 smithi129 ceph-mon[34885]: pgmap v292: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 Processor -- start 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 -- start start 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcb28e0 0x55ef500a0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b6460 con 0x55ef4fccb0a0 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b6b30 con 0x55ef4fcd3280 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f089628e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b7200 con 0x55ef4fcb28e0 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55226/0 (socket says 172.21.15.28:55226) 2024-09-21T05:45:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 -- 172.21.15.28:0/1664537478 learned_addr learned my addr 172.21.15.28:0/1664537478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088dab6640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcb28e0 0x55ef500a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 -- 172.21.15.28:0/1664537478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcb28e0 msgr2=0x55ef500a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcb28e0 0x55ef500a0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088e2b7640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 -- 172.21.15.28:0/1664537478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 msgr2=0x55ef500b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088d2b5640 1 -- 172.21.15.28:0/1664537478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef500a6510 con 0x55ef4fccb0a0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.879+0000 7f088e2b7640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f088d2b5640 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f08780aa920 tx=0x7f08780ace20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f070a053a29cdb67 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f08737fe640 1 -- 172.21.15.28:0/1664537478 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878095fb0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f08737fe640 1 -- 172.21.15.28:0/1664537478 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08780ad1c0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f08737fe640 1 -- 172.21.15.28:0/1664537478 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878096190 con 0x55ef4fccb0a0 2024-09-21T05:45:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f089628e740 1 -- 172.21.15.28:0/1664537478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 msgr2=0x55ef500b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.880+0000 7f089628e740 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f08780aa920 tx=0x7f08780ace20 comp rx=0 tx=0).stop 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 -- 172.21.15.28:0/1664537478 shutdown_connections 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fcd3280 0x55ef500b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fccb0a0 0x55ef500b1940 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 --2- 172.21.15.28:0/1664537478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcb28e0 0x55ef500a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 -- 172.21.15.28:0/1664537478 >> 172.21.15.28:0/1664537478 conn(0x55ef4fce8350 msgr2=0x55ef50092ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 -- 172.21.15.28:0/1664537478 shutdown_connections 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 -- 172.21.15.28:0/1664537478 wait complete. 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 -- start start 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.881+0000 7f089628e740 1 Processor -- start 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef500c0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 0x55ef500b9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b7fa0 con 0x55ef4fcb28e0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b78d0 con 0x55ef4fccb0a0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ef500b8670 con 0x55ef4fcd3280 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 10 client.0 _ll_register_callbacks cb 0x7f0884009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55236/0 (socket says 172.21.15.28:55236) 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 -- 172.21.15.28:0/3943351435 learned_addr learned my addr 172.21.15.28:0/3943351435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef500c0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088e2b7640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 0x55ef500b9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 -- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 msgr2=0x55ef500b9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 0x55ef500b9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 -- 172.21.15.28:0/3943351435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 msgr2=0x55ef500c0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef500c0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 -- 172.21.15.28:0/3943351435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef500ba200 con 0x55ef4fcb28e0 2024-09-21T05:45:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088e2b7640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 0x55ef500b9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f088dab6640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0884013790 tx=0x7f0884011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259058cab929f3dd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f08717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088400ec80 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f088400ee20 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.882+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08780ad1c0 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0884020020 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 10 client.15243 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef500bfbb0 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08780ad1c0 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f0884034030 con 0x55ef4fcb28e0 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 10 client.15243 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 10 client.15243 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T05:45:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 20 client.15243 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcf3f60 0x55ef501268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 4 client.15243 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "status"}) -- 0x55ef5000f610 con 0x55ef4fcf3f60 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.883+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcf3f60 0x55ef501268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f08717fa640 10 client.15243 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f088d2b5640 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcf3f60 0x55ef501268c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f34492d368f8774e server_cookie=b9efcfcb76a9ff3d in_seq=0 out_seq=0 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f08717fa640 1 -- 172.21.15.28:0/3943351435 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ef5000f610 con 0x55ef4fcf3f60 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f08717fa640 10 client.15243 handle_command_reply: tid=0 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 client.15243 shutdown 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 msgr2=0x55ef500c0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0884013790 tx=0x7f0884011c50 comp rx=0 tx=0).stop 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 shutdown_connections 2024-09-21T05:45:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55ef4fcf3f60 0x55ef501268c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ef4fcd3280 0x55ef500b9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4fccb0a0 0x55ef500c0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 --2- 172.21.15.28:0/3943351435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef4fcb28e0 0x55ef500c0610 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f08717fa640 0 client.15243 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.884+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 >> 172.21.15.28:0/3943351435 conn(0x55ef4fce8350 msgr2=0x55ef500a0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 shutdown_connections 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f089628e740 1 -- 172.21.15.28:0/3943351435 wait complete. 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f089628e740 20 client.15243 trim_cache size 0 max 16384 2024-09-21T05:45:56.895 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24664, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 66.697256472000007, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 103.077659862 2024-09-21T05:45:56.896 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 msgr2=0x7f08740b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 0x7f08740b8bb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0880005cc0 tx=0x7f0880005c30 comp rx=0 tx=0).stop 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 msgr2=0x7f088819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f087800c920 tx=0x7f087800cde0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 shutdown_connections 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888110200 0x7f08881a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f08740b66a0 0x7f08740b8bb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0888102fa0 0x7f088819cbb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 --2- 172.21.15.28:0/3418887140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888102650 0x7f088819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 >> 172.21.15.28:0/3418887140 conn(0x7f0888079760 msgr2=0x7f0888108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 shutdown_connections 2024-09-21T05:45:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:56.885+0000 7f088fd1a640 1 -- 172.21.15.28:0/3418887140 wait complete. 2024-09-21T05:45:57.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T05:45:57.911 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:58.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/2333783007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a0108cf0 msgr2=0x7fa5a010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/2333783007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a0108cf0 0x7fa5a010f520 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa598009ca0 tx=0x7fa59802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/2333783007 shutdown_connections 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/2333783007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a0108cf0 0x7fa5a010f520 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/2333783007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a01083a0 0x7fa5a01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/2333783007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 0x7fa5a0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/2333783007 >> 172.21.15.28:0/2333783007 conn(0x7fa5a0079730 msgr2=0x7fa5a0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/2333783007 shutdown_connections 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.391+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/2333783007 wait complete. 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 Processor -- start 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 -- start start 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 0x7fa5a019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0108cf0 0x7fa5a019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a0112960 con 0x7fa5a01083a0 2024-09-21T05:45:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a01127e0 con 0x7fa5a0108cf0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a7b5c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a0112ae0 con 0x7fa5a0102a60 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a50f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a50f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55254/0 (socket says 172.21.15.28:55254) 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a50f7640 1 -- 172.21.15.28:0/3491529117 learned_addr learned my addr 172.21.15.28:0/3491529117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.392+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0108cf0 0x7fa5a019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 0x7fa5a019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 -- 172.21.15.28:0/3491529117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 msgr2=0x7fa5a019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 0x7fa5a019a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 -- 172.21.15.28:0/3491529117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0108cf0 msgr2=0x7fa5a019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0108cf0 0x7fa5a019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 -- 172.21.15.28:0/3491529117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a01a0310 con 0x7fa5a01083a0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa59000d770 tx=0x7fa59000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa596ffd640 1 -- 172.21.15.28:0/3491529117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590017070 con 0x7fa5a01083a0 2024-09-21T05:45:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa596ffd640 1 -- 172.21.15.28:0/3491529117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa590009c30 con 0x7fa5a01083a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a01a0600 con 0x7fa5a01083a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa596ffd640 1 -- 172.21.15.28:0/3491529117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59000f620 con 0x7fa5a01083a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.393+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a01a6d60 con 0x7fa5a01083a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.395+0000 7fa5ae0d0740 1 Processor -- start 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 -- start start 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa596ffd640 1 -- 172.21.15.28:0/3491529117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fa59001a070 con 0x7fa5a01083a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad25cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad25d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652ad1d4c10 con 0x5652ace8e0a0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652ad1c4040 con 0x5652ace96280 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5652ad1c7e50 con 0x5652aceb6f60 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55262/0 (socket says 172.21.15.28:55262) 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2366495366 learned_addr learned my addr 172.21.15.28:0/2366495366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad25cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad25d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2366495366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 msgr2=0x5652ad25cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad25cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2366495366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 msgr2=0x5652ad25d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad25d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2366495366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652acf22540 con 0x5652ace8e0a0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.396+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad25d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009bc0 tx=0x7fa59c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05fe1d2cc49a639 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa58bfff640 1 -- 172.21.15.28:0/2366495366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c016070 con 0x5652ace8e0a0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa596ffd640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 0x7fa58c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa58bfff640 1 -- 172.21.15.28:0/2366495366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa59c0054d0 con 0x5652ace8e0a0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 0x7fa58c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa596ffd640 1 -- 172.21.15.28:0/3491529117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa5900e7200 con 0x7fa5a01083a0 2024-09-21T05:45:58.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.397+0000 7fa58bfff640 1 -- 172.21.15.28:0/2366495366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c011470 con 0x5652ace8e0a0 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2366495366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 msgr2=0x5652ad25d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 0x7fa58c0b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa5980093f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009bc0 tx=0x7fa59c00e6c0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2366495366 shutdown_connections 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad25d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad25d440 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2366495366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad25cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2366495366 >> 172.21.15.28:0/2366495366 conn(0x5652aceab350 msgr2=0x5652ad25adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2366495366 shutdown_connections 2024-09-21T05:45:58.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.398+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2366495366 wait complete. 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 -- start start 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 Processor -- start 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad268c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad274a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652ad0255f0 con 0x5652ace8e0a0 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652ad15f010 con 0x5652ace96280 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5652ad26b730 con 0x5652aceb6f60 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5ae0d0740 10 client.0 _ll_register_callbacks cb 0x5652ad1be340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5a58f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5a58f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55274/0 (socket says 172.21.15.28:55274) 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.399+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2074149554 learned_addr learned my addr 172.21.15.28:0/2074149554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad274a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad268c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 msgr2=0x5652ad274a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad274a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2074149554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 msgr2=0x5652ad268c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad268c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 -- 172.21.15.28:0/2074149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652ad18ca00 con 0x5652ace8e0a0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad274a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad268c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0048f0 tx=0x7fa59c004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ea82ca5eb9c70c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c028060 con 0x5652ace8e0a0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa59c004ee0 con 0x5652ace8e0a0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59c0054d0 con 0x5652ace8e0a0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa59c016040 con 0x5652ace8e0a0 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5ae0d0740 10 client.15255 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:58.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5652ad277500 con 0x5652ace8e0a0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.400+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa59c0054d0 con 0x5652ace8e0a0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fa59c00f040 con 0x5652ace8e0a0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 10 client.15255 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 10 client.15255 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 20 client.15255 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace758e0 0x5652ad27abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 4 client.15255 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652ad1d6b60 con 0x5652ace758e0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.401+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace758e0 0x5652ad27abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.402+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace758e0 0x5652ad27abf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee28cb0ab7b4d6f6 server_cookie=161f71aa7c417321 in_seq=0 out_seq=0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.402+0000 7fa5897fa640 10 client.15255 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.402+0000 7fa5897fa640 1 -- 172.21.15.28:0/2074149554 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5652ad1d6b60 con 0x5652ace758e0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.402+0000 7fa5897fa640 10 client.15255 handle_command_reply: tid=0 2024-09-21T05:45:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.402+0000 7fa5ae0d0740 1 client.15255 shutdown 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 msgr2=0x5652ad268700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0048f0 tx=0x7fa59c004920 comp rx=0 tx=0).stop 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 shutdown_connections 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace758e0 0x5652ad27abf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad274a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace96280 0x5652ad268c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/2074149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace8e0a0 0x5652ad268700 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5897fa640 0 client.15255 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 >> 172.21.15.28:0/2074149554 conn(0x5652aceab350 msgr2=0x5652ad25e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 shutdown_connections 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/2074149554 wait complete. 2024-09-21T05:45:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.403+0000 7fa5ae0d0740 20 client.15255 trim_cache size 0 max 16384 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 Processor -- start 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 -- start start 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad263840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad266f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652ad278f30 con 0x5652ace96280 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652ad279600 con 0x5652ace8e0a0 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5652ad279cd0 con 0x5652aceb6f60 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5a50f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5a50f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55276/0 (socket says 172.21.15.28:55276) 2024-09-21T05:45:58.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.421+0000 7fa5a50f7640 1 -- 172.21.15.28:0/1019627921 learned_addr learned my addr 172.21.15.28:0/1019627921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad266f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad263840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 -- 172.21.15.28:0/1019627921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 msgr2=0x5652ad266f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad266f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 -- 172.21.15.28:0/1019627921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 msgr2=0x5652ad263840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad263840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 -- 172.21.15.28:0/1019627921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652ad25bc20 con 0x5652ace96280 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad263840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5900a9700 tx=0x7fa5900e96f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4c6cdc55cdc0df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa58b7fe640 1 -- 172.21.15.28:0/1019627921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590021040 con 0x5652ace96280 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa58b7fe640 1 -- 172.21.15.28:0/1019627921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa590009de0 con 0x5652ace96280 2024-09-21T05:45:58.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.422+0000 7fa58b7fe640 1 -- 172.21.15.28:0/1019627921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5900abe00 con 0x5652ace96280 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1019627921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 msgr2=0x5652ad263d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5900a9700 tx=0x7fa5900e96f0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1019627921 shutdown_connections 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652aceb6f60 0x5652ad266f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652ace96280 0x5652ad263d80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1019627921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad263840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1019627921 >> 172.21.15.28:0/1019627921 conn(0x5652aceab350 msgr2=0x5652ad262ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1019627921 shutdown_connections 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1019627921 wait complete. 2024-09-21T05:45:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.423+0000 7fa5ae0d0740 1 -- start start 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 Processor -- start 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad283850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652ace96280 0x5652ad27cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652ad27aa70 con 0x5652aceb6f60 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652ad27a3a0 con 0x5652ace8e0a0 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5652ad27b140 con 0x5652ace96280 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5ae0d0740 10 client.0 _ll_register_callbacks cb 0x7fa59c009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55280/0 (socket says 172.21.15.28:55280) 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 -- 172.21.15.28:0/1024391871 learned_addr learned my addr 172.21.15.28:0/1024391871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad283850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a50f7640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652ace96280 0x5652ad27cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 -- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652ace96280 msgr2=0x5652ad27cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652ace96280 0x5652ad27cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.424+0000 7fa5a60f9640 1 -- 172.21.15.28:0/1024391871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 msgr2=0x5652ad283850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad283850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5a60f9640 1 -- 172.21.15.28:0/1024391871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652ad282490 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5a60f9640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5980316d0 tx=0x7fa59803f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bdf4401ae71e98 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598055060 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa598004240 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa590009de0 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa598004440 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5ae0d0740 10 client.15261 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5652ad2e6180 con 0x5652aceb6f60 2024-09-21T05:45:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa590009de0 con 0x5652aceb6f60 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.425+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fa598044040 con 0x5652aceb6f60 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 10 client.15261 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 10 client.15261 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 20 client.15261 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace4ac90 0x5652ad2e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 4 client.15261 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "status"}) -- 0x5652ad1d2610 con 0x5652ace4ac90 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace4ac90 0x5652ad2e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5897fa640 10 client.15261 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5a58f8640 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace4ac90 0x5652ad2e9a80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a92d5e941e85cd81 server_cookie=d2fa9200b5282031 in_seq=0 out_seq=0 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5897fa640 1 -- 172.21.15.28:0/1024391871 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5652ad1d2610 con 0x5652ace4ac90 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5897fa640 10 client.15261 handle_command_reply: tid=0 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.426+0000 7fa5ae0d0740 1 client.15261 shutdown 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 msgr2=0x5652ad27d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5980316d0 tx=0x7fa59803f040 comp rx=0 tx=0).stop 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 shutdown_connections 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5652ace4ac90 0x5652ad2e9a80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652aceb6f60 0x5652ad27d410 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5652ace96280 0x5652ad27cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 --2- 172.21.15.28:0/1024391871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652ace8e0a0 0x5652ad283850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5897fa640 0 client.15261 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 >> 172.21.15.28:0/1024391871 conn(0x5652aceab350 msgr2=0x5652ad2684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 shutdown_connections 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 1 -- 172.21.15.28:0/1024391871 wait complete. 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5ae0d0740 20 client.15261 trim_cache size 0 max 16384 2024-09-21T05:45:58.433 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24673, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 69.247605702000001, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 102.241086413 2024-09-21T05:45:58.434 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 msgr2=0x7fa58c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 0x7fa58c0b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa5980093f0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 msgr2=0x7fa5a019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.427+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa59000d770 tx=0x7fa59000dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 shutdown_connections 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0108cf0 0x7fa5a019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa58c0b65f0 0x7fa58c0b8b00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5a01083a0 0x7fa5a019a940 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 --2- 172.21.15.28:0/3491529117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5a0102a60 0x7fa5a019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:58.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 >> 172.21.15.28:0/3491529117 conn(0x7fa5a0079730 msgr2=0x7fa5a006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:58.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 shutdown_connections 2024-09-21T05:45:58.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:58.428+0000 7fa5a7b5c640 1 -- 172.21.15.28:0/3491529117 wait complete. 2024-09-21T05:45:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:45:58 smithi028 ceph-mon[31227]: pgmap v293: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:58.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:45:58 smithi181 ceph-mon[34819]: pgmap v293: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:45:58 smithi129 ceph-mon[34885]: pgmap v293: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:45:59.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T05:45:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:45:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 -- 172.21.15.28:0/442165448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63601094a0 msgr2=0x7f63601098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 --2- 172.21.15.28:0/442165448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63601094a0 0x7f63601098b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6348009900 tx=0x7f634802a410 comp rx=0 tx=0).stop 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 -- 172.21.15.28:0/442165448 shutdown_connections 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 --2- 172.21.15.28:0/442165448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 0x7f6360110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 --2- 172.21.15.28:0/442165448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63601094a0 0x7f63601098b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 --2- 172.21.15.28:0/442165448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6360103b60 0x7f6360103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 -- 172.21.15.28:0/442165448 >> 172.21.15.28:0/442165448 conn(0x7f636007b850 msgr2=0x7f636007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 -- 172.21.15.28:0/442165448 shutdown_connections 2024-09-21T05:45:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.897+0000 7f636685e640 1 -- 172.21.15.28:0/442165448 wait complete. 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.898+0000 7f636685e640 1 Processor -- start 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.898+0000 7f636685e640 1 -- start start 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.898+0000 7f636685e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63601094a0 0x7f636019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 0x7f63601a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6360113ab0 con 0x7f6360103b60 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6360113930 con 0x7f6360109df0 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6360113c30 con 0x7f63601094a0 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55306/0 (socket says 172.21.15.28:55306) 2024-09-21T05:45:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 -- 172.21.15.28:0/310396625 learned_addr learned my addr 172.21.15.28:0/310396625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f6357fff640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63601094a0 0x7f636019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f6364dfb640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 0x7f63601a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 -- 172.21.15.28:0/310396625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63601094a0 msgr2=0x7f636019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63601094a0 0x7f636019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 -- 172.21.15.28:0/310396625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 msgr2=0x7f63601a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 0x7f63601a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 -- 172.21.15.28:0/310396625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63601a26d0 con 0x7f6360103b60 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635ffff640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f635000d770 tx=0x7f635000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635dffb640 1 -- 172.21.15.28:0/310396625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6350017070 con 0x7f6360103b60 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635dffb640 1 -- 172.21.15.28:0/310396625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63500054d0 con 0x7f6360103b60 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f635dffb640 1 -- 172.21.15.28:0/310396625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6350004c30 con 0x7f6360103b60 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.899+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63601a29c0 con 0x7f6360103b60 2024-09-21T05:45:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.900+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63601a9120 con 0x7f6360103b60 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.902+0000 7f636cdd2740 1 Processor -- start 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.902+0000 7f636cdd2740 1 -- start start 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222817f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222818440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b22278fc10 con 0x55b222451280 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b22277f040 con 0x55b222471f60 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f636cdd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b222782e50 con 0x55b2224490a0 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f635dffb640 1 -- 172.21.15.28:0/310396625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f6350010460 con 0x7f6360103b60 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55316/0 (socket says 172.21.15.28:55316) 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 -- 172.21.15.28:0/1127293255 learned_addr learned my addr 172.21.15.28:0/1127293255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6364dfb640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222817f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f635ffff640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222818440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 -- 172.21.15.28:0/1127293255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 msgr2=0x55b222818440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222818440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 -- 172.21.15.28:0/1127293255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 msgr2=0x55b222817f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222817f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 -- 172.21.15.28:0/1127293255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2224dd540 con 0x55b222451280 2024-09-21T05:45:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f635ffff640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222818440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:45:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.903+0000 7f6357fff640 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6348000c00 tx=0x7f6348032c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e8704052a3e7d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f63567fc640 1 -- 172.21.15.28:0/1127293255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634803f070 con 0x55b222451280 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f63567fc640 1 -- 172.21.15.28:0/1127293255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6348032e40 con 0x55b222451280 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f63567fc640 1 -- 172.21.15.28:0/1127293255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348035070 con 0x55b222451280 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f635dffb640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 0x7f634c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f6364dfb640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 0x7f634c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.904+0000 7f635dffb640 1 -- 172.21.15.28:0/310396625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f63500e71f0 con 0x7f6360103b60 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- 172.21.15.28:0/1127293255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 msgr2=0x55b222818980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6348000c00 tx=0x7f6348032c50 comp rx=0 tx=0).stop 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- 172.21.15.28:0/1127293255 shutdown_connections 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222818980 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222818440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 --2- 172.21.15.28:0/1127293255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222817f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- 172.21.15.28:0/1127293255 >> 172.21.15.28:0/1127293255 conn(0x55b222466350 msgr2=0x55b222815df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f6364dfb640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 0x7f634c0b8b00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6358005fd0 tx=0x7f63580095a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- 172.21.15.28:0/1127293255 shutdown_connections 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- 172.21.15.28:0/1127293255 wait complete. 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.905+0000 7f636cdd2740 1 -- start start 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 Processor -- start 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222823980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b22282f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2225e05f0 con 0x55b222451280 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b22271a010 con 0x55b222471f60 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b222826680 con 0x55b2224490a0 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f636cdd2740 10 client.0 _ll_register_callbacks cb 0x55b222779340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55332/0 (socket says 172.21.15.28:55332) 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 -- 172.21.15.28:0/3905162346 learned_addr learned my addr 172.21.15.28:0/3905162346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f635ffff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222823980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6364dfb640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b22282f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 -- 172.21.15.28:0/3905162346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 msgr2=0x55b222823980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222823980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 -- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 msgr2=0x55b22282f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b22282f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6357fff640 1 -- 172.21.15.28:0/3905162346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b222747a00 con 0x55b222451280 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.906+0000 7f6364dfb640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b22282f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f6357fff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6348006010 tx=0x7f63480316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ad3cbacc27f656 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634804b060 con 0x55b222451280 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b222791b60 con 0x55b222451280 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6348004230 con 0x55b222451280 2024-09-21T05:45:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f634803f070 con 0x55b222451280 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f636cdd2740 10 client.15273 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b2228324f0 con 0x55b222451280 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b222791b60 con 0x55b222451280 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f6348003710 con 0x55b222451280 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f636cdd2740 10 client.15273 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.907+0000 7f636cdd2740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f636cdd2740 20 client.15273 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b2224308e0 0x55b222835ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f636cdd2740 4 client.15273 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b22278d610 con 0x55b2224308e0 2024-09-21T05:45:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f635ffff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b2224308e0 0x55b222835ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f635ffff640 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b2224308e0 0x55b222835ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=886fb17118187528 server_cookie=1db3813af104ec2c in_seq=0 out_seq=0 2024-09-21T05:45:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.908+0000 7f633f7fe640 10 client.15273 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:45:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f633f7fe640 1 -- 172.21.15.28:0/3905162346 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b22278d610 con 0x55b2224308e0 2024-09-21T05:45:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f633f7fe640 10 client.15273 handle_command_reply: tid=0 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f636cdd2740 1 client.15273 shutdown 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 msgr2=0x55b222823ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6348006010 tx=0x7f63480316d0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 shutdown_connections 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.909+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b2224308e0 0x55b222835ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b22282f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b222823ec0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3905162346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b2224490a0 0x55b222823980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 >> 172.21.15.28:0/3905162346 conn(0x55b222466350 msgr2=0x55b22281b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f633f7fe640 0 client.15273 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 shutdown_connections 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 1 -- 172.21.15.28:0/3905162346 wait complete. 2024-09-21T05:45:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.910+0000 7f636cdd2740 20 client.15273 trim_cache size 0 max 16384 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 Processor -- start 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 -- start start 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 0x55b22281e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222815d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b222833df0 con 0x55b2224490a0 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b2228344c0 con 0x55b222471f60 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f636cdd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b222834b90 con 0x55b222451280 2024-09-21T05:45:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f635ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f635ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55338/0 (socket says 172.21.15.28:55338) 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.928+0000 7f635ffff640 1 -- 172.21.15.28:0/3400662067 learned_addr learned my addr 172.21.15.28:0/3400662067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6364dfb640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222815d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6357fff640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 0x55b22281e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 -- 172.21.15.28:0/3400662067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 msgr2=0x55b22281e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 0x55b22281e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 -- 172.21.15.28:0/3400662067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 msgr2=0x55b222815d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222815d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 -- 172.21.15.28:0/3400662067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b22281f010 con 0x55b2224490a0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6357fff640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 0x55b22281e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f635ffff640 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63500e4ad0 tx=0x7f63500e96e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fa9541b2d1af6b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6355ffb640 1 -- 172.21.15.28:0/3400662067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6350098020 con 0x55b2224490a0 2024-09-21T05:45:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6355ffb640 1 -- 172.21.15.28:0/3400662067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6350005680 con 0x55b2224490a0 2024-09-21T05:45:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.929+0000 7f6355ffb640 1 -- 172.21.15.28:0/3400662067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6350020040 con 0x55b2224490a0 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- 172.21.15.28:0/3400662067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 msgr2=0x55b222834640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63500e4ad0 tx=0x7f63500e96e0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- 172.21.15.28:0/3400662067 shutdown_connections 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b222471f60 0x55b222815d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222451280 0x55b22281e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 --2- 172.21.15.28:0/3400662067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2224490a0 0x55b222834640 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- 172.21.15.28:0/3400662067 >> 172.21.15.28:0/3400662067 conn(0x55b222466350 msgr2=0x55b222815600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- 172.21.15.28:0/3400662067 shutdown_connections 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- 172.21.15.28:0/3400662067 wait complete. 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.930+0000 7f636cdd2740 1 -- start start 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 Processor -- start 2024-09-21T05:45:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2224490a0 0x55b22283e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222471f60 0x55b222837c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b222835930 con 0x55b222451280 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b222835260 con 0x55b2224490a0 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b222836000 con 0x55b222471f60 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f636cdd2740 10 client.0 _ll_register_callbacks cb 0x7f6358009440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55340/0 (socket says 172.21.15.28:55340) 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 -- 172.21.15.28:0/2697045013 learned_addr learned my addr 172.21.15.28:0/2697045013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f635ffff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2224490a0 0x55b22283e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 -- 172.21.15.28:0/2697045013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222471f60 msgr2=0x55b222837c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222471f60 0x55b222837c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 -- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2224490a0 msgr2=0x55b22283e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:45:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2224490a0 0x55b22283e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.931+0000 7f6357fff640 1 -- 172.21.15.28:0/2697045013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b222838160 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f6357fff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6348003c70 tx=0x7f6348032a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edc32abb9951b70 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634804b070 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6348004100 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6350005680 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6348035040 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 10 client.15279 fetch_fsmap learned FSMap version 26 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b22283da00 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6350005680 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f634805f030 con 0x55b222451280 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 10 client.15279 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T05:45:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 20 client.15279 populate_metadata read hostname 'smithi028' 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b222405c90 0x55b2228a4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 4 client.15279 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.932+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "status"}) -- 0x55b22278aa40 con 0x55b222405c90 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f635ffff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b222405c90 0x55b2228a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f633f7fe640 10 client.15279 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f635ffff640 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b222405c90 0x55b2228a4410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=281396a29bbf5ee1 server_cookie=8ff64516228c9992 in_seq=0 out_seq=0 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f633f7fe640 1 -- 172.21.15.28:0/2697045013 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b22278aa40 con 0x55b222405c90 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f633f7fe640 10 client.15279 handle_command_reply: tid=0 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f636cdd2740 1 client.15279 shutdown 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 msgr2=0x55b2228376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.933+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6348003c70 tx=0x7f6348032a80 comp rx=0 tx=0).stop 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 shutdown_connections 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55b222405c90 0x55b2228a4410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b222471f60 0x55b222837c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b222451280 0x55b2228376e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 --2- 172.21.15.28:0/2697045013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2224490a0 0x55b22283e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 >> 172.21.15.28:0/2697045013 conn(0x55b222466350 msgr2=0x55b2228143b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f633f7fe640 0 client.15279 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 shutdown_connections 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 1 -- 172.21.15.28:0/2697045013 wait complete. 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636cdd2740 20 client.15279 trim_cache size 0 max 16384 2024-09-21T05:45:59.940 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24680, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 71.764164637999997, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 90.674959713999996 2024-09-21T05:45:59.941 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 msgr2=0x7f634c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 0x7f634c0b8b00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6358005fd0 tx=0x7f63580095a0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 msgr2=0x7f636019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f635000d770 tx=0x7f635000dc30 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 shutdown_connections 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360109df0 0x7f63601a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63601094a0 0x7f636019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f634c0b65f0 0x7f634c0b8b00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 --2- 172.21.15.28:0/310396625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6360103b60 0x7f636019c760 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:45:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 >> 172.21.15.28:0/310396625 conn(0x7f636007b850 msgr2=0x7f63601073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:45:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.934+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 shutdown_connections 2024-09-21T05:45:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:45:59.935+0000 7f636685e640 1 -- 172.21.15.28:0/310396625 wait complete. 2024-09-21T05:46:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:00 smithi028 ceph-mon[31227]: pgmap v294: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:00.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T05:46:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:00 smithi181 ceph-mon[34819]: pgmap v294: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:00 smithi129 ceph-mon[34885]: pgmap v294: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 -- 172.21.15.28:0/1249122364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214108d20 msgr2=0x7f521410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 --2- 172.21.15.28:0/1249122364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214108d20 0x7f521410f550 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f520c009900 tx=0x7f520c02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 -- 172.21.15.28:0/1249122364 shutdown_connections 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 --2- 172.21.15.28:0/1249122364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214108d20 0x7f521410f550 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 --2- 172.21.15.28:0/1249122364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f52141087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 --2- 172.21.15.28:0/1249122364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214102ae0 0x7f5214102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 -- 172.21.15.28:0/1249122364 >> 172.21.15.28:0/1249122364 conn(0x7f5214079760 msgr2=0x7f5214079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 -- 172.21.15.28:0/1249122364 shutdown_connections 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.499+0000 7f52198fb640 1 -- 172.21.15.28:0/1249122364 wait complete. 2024-09-21T05:46:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 Processor -- start 2024-09-21T05:46:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 -- start start 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f521419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214108d20 0x7f52141a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5214112990 con 0x7f5214102ae0 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5214112810 con 0x7f5214108d20 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f52198fb640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5214112b10 con 0x7f52141083d0 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f5212ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f5212ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55368/0 (socket says 172.21.15.28:55368) 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.500+0000 7f5212ffd640 1 -- 172.21.15.28:0/1621754639 learned_addr learned my addr 172.21.15.28:0/1621754639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52137fe640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214108d20 0x7f52141a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52127fc640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f521419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 -- 172.21.15.28:0/1621754639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 msgr2=0x7f521419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f521419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 -- 172.21.15.28:0/1621754639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214108d20 msgr2=0x7f52141a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214108d20 0x7f52141a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 -- 172.21.15.28:0/1621754639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52141a2490 con 0x7f5214102ae0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52127fc640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f521419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f5212ffd640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f520400eab0 tx=0x7f520400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52188f9640 1 -- 172.21.15.28:0/1621754639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520400ce40 con 0x7f5214102ae0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52188f9640 1 -- 172.21.15.28:0/1621754639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5204011920 con 0x7f5214102ae0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52188f9640 1 -- 172.21.15.28:0/1621754639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204010a30 con 0x7f5214102ae0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52141a2780 con 0x7f5214102ae0 2024-09-21T05:46:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.501+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52141a8fa0 con 0x7f5214102ae0 2024-09-21T05:46:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.504+0000 7f521fe6f740 1 Processor -- start 2024-09-21T05:46:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.504+0000 7f521fe6f740 1 -- start start 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43fee1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43fee1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f43fe58c10 con 0x55f43fb3af60 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f43fe48040 con 0x55f43fb1a280 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f521fe6f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f43fe4be50 con 0x55f43fb120a0 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52188f9640 1 -- 172.21.15.28:0/1621754639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f5204010bd0 con 0x7f5214102ae0 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55374/0 (socket says 172.21.15.28:55374) 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 -- 172.21.15.28:0/3701509677 learned_addr learned my addr 172.21.15.28:0/3701509677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52127fc640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43fee1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f5212ffd640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43fee1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 -- 172.21.15.28:0/3701509677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 msgr2=0x55f43fee1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43fee1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 -- 172.21.15.28:0/3701509677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 msgr2=0x55f43fee1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43fee1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f52137fe640 1 -- 172.21.15.28:0/3701509677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f43fba6540 con 0x55f43fb3af60 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.505+0000 7f5212ffd640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43fee1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f52137fe640 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f520c009900 tx=0x7f520c032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8969ff9ff6751cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f51f9ffb640 1 -- 172.21.15.28:0/3701509677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c03f070 con 0x55f43fb3af60 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f51f9ffb640 1 -- 172.21.15.28:0/3701509677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f520c0316a0 con 0x55f43fb3af60 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f51f9ffb640 1 -- 172.21.15.28:0/3701509677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c035070 con 0x55f43fb3af60 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f52188f9640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 0x7f51fc0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f5212ffd640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 0x7f51fc0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.506+0000 7f52188f9640 1 -- 172.21.15.28:0/1621754639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f52040e7cd0 con 0x7f5214102ae0 2024-09-21T05:46:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- 172.21.15.28:0/3701509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 msgr2=0x55f43fee0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f5212ffd640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 0x7f51fc0b8bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f52040aae10 tx=0x7f52040104d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f520c009900 tx=0x7f520c032e70 comp rx=0 tx=0).stop 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- 172.21.15.28:0/3701509677 shutdown_connections 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43fee1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43fee1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 --2- 172.21.15.28:0/3701509677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fee0f00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- 172.21.15.28:0/3701509677 >> 172.21.15.28:0/3701509677 conn(0x55f43fb2f350 msgr2=0x55f43fededf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- 172.21.15.28:0/3701509677 shutdown_connections 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- 172.21.15.28:0/3701509677 wait complete. 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.507+0000 7f521fe6f740 1 -- start start 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 Processor -- start 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43feeca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43feecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f43fca95f0 con 0x55f43fb3af60 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f43fde3010 con 0x55f43fb1a280 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f43feef680 con 0x55f43fb120a0 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f521fe6f740 10 client.0 _ll_register_callbacks cb 0x55f43fe42340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55382/0 (socket says 172.21.15.28:55382) 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f52137fe640 1 -- 172.21.15.28:0/1639686532 learned_addr learned my addr 172.21.15.28:0/1639686532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f52137fe640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43feeca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.508+0000 7f52127fc640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43feecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 -- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 msgr2=0x55f43feeca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43feeca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 -- 172.21.15.28:0/1639686532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 msgr2=0x55f43feecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43feecf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 -- 172.21.15.28:0/1639686532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f43fe10a00 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52127fc640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43feecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f52137fe640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f520c0057c0 tx=0x7f520c0053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6c7e783c17aaec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f51f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c050060 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f520c005480 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f520c0316a0 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f520c03f040 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f521fe6f740 10 client.15291 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f43fefb610 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.509+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f520c0316a0 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f520c0039c0 con 0x55f43fb3af60 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 10 client.15291 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 10 client.15291 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 20 client.15291 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43faf98e0 0x55f43fefebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 4 client.15291 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f43fe5ab60 con 0x55f43faf98e0 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.510+0000 7f52127fc640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43faf98e0 0x55f43fefebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.511+0000 7f52127fc640 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43faf98e0 0x55f43fefebd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79fc59ace4d73861 server_cookie=b7dc0365bb58f745 in_seq=0 out_seq=0 2024-09-21T05:46:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.511+0000 7f51f2ffd640 10 client.15291 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.511+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1639686532 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f43fe5ab60 con 0x55f43faf98e0 2024-09-21T05:46:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.511+0000 7f51f2ffd640 10 client.15291 handle_command_reply: tid=0 2024-09-21T05:46:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 client.15291 shutdown 2024-09-21T05:46:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 msgr2=0x55f43fef8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f520c0057c0 tx=0x7f520c0053a0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 shutdown_connections 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43faf98e0 0x55f43fefebd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fef8970 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43feecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1639686532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43feeca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f51f2ffd640 0 client.15291 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 >> 172.21.15.28:0/1639686532 conn(0x55f43fb2f350 msgr2=0x55f43fee40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 shutdown_connections 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 1 -- 172.21.15.28:0/1639686532 wait complete. 2024-09-21T05:46:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.512+0000 7f521fe6f740 20 client.15291 trim_cache size 0 max 16384 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.530+0000 7f521fe6f740 1 Processor -- start 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.530+0000 7f521fe6f740 1 -- start start 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb120a0 0x55f43fef88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb1a280 0x55f43fef8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f43fefcf10 con 0x55f43fb3af60 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f43fefd5e0 con 0x55f43fb120a0 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f521fe6f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f43fefdcb0 con 0x55f43fb1a280 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55390/0 (socket says 172.21.15.28:55390) 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 -- 172.21.15.28:0/1458091970 learned_addr learned my addr 172.21.15.28:0/1458091970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb120a0 0x55f43fef88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 -- 172.21.15.28:0/1458091970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb1a280 msgr2=0x55f43fef8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb1a280 0x55f43fef8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 -- 172.21.15.28:0/1458091970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb120a0 msgr2=0x55f43fef88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb120a0 0x55f43fef88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 -- 172.21.15.28:0/1458091970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f43feedaf0 con 0x55f43fb3af60 2024-09-21T05:46:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f52137fe640 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f520c032dc0 tx=0x7f520c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8feba4cb2f9ba26 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f51f97fa640 1 -- 172.21.15.28:0/1458091970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c050030 con 0x55f43fb3af60 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f51f97fa640 1 -- 172.21.15.28:0/1458091970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f520c032e70 con 0x55f43fb3af60 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.531+0000 7f51f97fa640 1 -- 172.21.15.28:0/1458091970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c047070 con 0x55f43fb3af60 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 -- 172.21.15.28:0/1458091970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 msgr2=0x55f43fede600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f520c032dc0 tx=0x7f520c02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 -- 172.21.15.28:0/1458091970 shutdown_connections 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43fede600 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb1a280 0x55f43fef8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1458091970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb120a0 0x55f43fef88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 -- 172.21.15.28:0/1458091970 >> 172.21.15.28:0/1458091970 conn(0x55f43fb2f350 msgr2=0x55f43fee79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 -- 172.21.15.28:0/1458091970 shutdown_connections 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.532+0000 7f521fe6f740 1 -- 172.21.15.28:0/1458091970 wait complete. 2024-09-21T05:46:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 -- start start 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 Processor -- start 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43ff076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43ff00ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f43fefea50 con 0x55f43fb3af60 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f43fefe380 con 0x55f43fb1a280 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f43feff120 con 0x55f43fb120a0 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f521fe6f740 10 client.0 _ll_register_callbacks cb 0x7f520800b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f52137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55400/0 (socket says 172.21.15.28:55400) 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.533+0000 7f52137fe640 1 -- 172.21.15.28:0/1425129975 learned_addr learned my addr 172.21.15.28:0/1425129975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43ff076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52127fc640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43ff00ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 -- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 msgr2=0x55f43ff076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43ff076a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 -- 172.21.15.28:0/1425129975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 msgr2=0x55f43ff00ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43ff00ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 -- 172.21.15.28:0/1425129975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f43ff018b0 con 0x55f43fb3af60 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f52137fe640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f520c0096c0 tx=0x7f520c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428de16e575fbea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f51f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c067060 con 0x55f43fb3af60 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f43fe56610 con 0x55f43fb3af60 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f520c004110 con 0x55f43fb3af60 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f520c047070 con 0x55f43fb3af60 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f521fe6f740 10 client.15297 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f43ff6a0a0 con 0x55f43fb3af60 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.534+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f43fe56610 con 0x55f43fb3af60 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f520c054440 con 0x55f43fb3af60 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 10 client.15297 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 10 client.15297 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 20 client.15297 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43facec90 0x55f43ff6d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 4 client.15297 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "status"}) -- 0x55f43fe53a40 con 0x55f43facec90 2024-09-21T05:46:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.535+0000 7f52127fc640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43facec90 0x55f43ff6d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f51f2ffd640 10 client.15297 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f52127fc640 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43facec90 0x55f43ff6d500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bcabe88881bad96 server_cookie=277cf43ddb1abe14 in_seq=0 out_seq=0 2024-09-21T05:46:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f51f2ffd640 1 -- 172.21.15.28:0/1425129975 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f43fe53a40 con 0x55f43facec90 2024-09-21T05:46:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f51f2ffd640 10 client.15297 handle_command_reply: tid=0 2024-09-21T05:46:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 client.15297 shutdown 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 msgr2=0x55f43ff01120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f520c0096c0 tx=0x7f520c005c90 comp rx=0 tx=0).stop 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 shutdown_connections 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f43facec90 0x55f43ff6d500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f43fb3af60 0x55f43ff01120 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f43fb1a280 0x55f43ff00ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 --2- 172.21.15.28:0/1425129975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f43fb120a0 0x55f43ff076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 >> 172.21.15.28:0/1425129975 conn(0x55f43fb2f350 msgr2=0x55f43fede600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f51f2ffd640 0 client.15297 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 shutdown_connections 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 1 -- 172.21.15.28:0/1425129975 wait complete. 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.536+0000 7f521fe6f740 20 client.15297 trim_cache size 0 max 16384 2024-09-21T05:46:01.542 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24688, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 81.413528678999995, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 100.499025828 2024-09-21T05:46:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 msgr2=0x7f51fc0b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 0x7f51fc0b8bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f52040aae10 tx=0x7f52040104d0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 msgr2=0x7f521419c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f520400eab0 tx=0x7f520400ef70 comp rx=0 tx=0).stop 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 shutdown_connections 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5214108d20 0x7f52141a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52141083d0 0x7f521419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51fc0b66a0 0x7f51fc0b8bb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 --2- 172.21.15.28:0/1621754639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214102ae0 0x7f521419c5e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 >> 172.21.15.28:0/1621754639 conn(0x7f5214079760 msgr2=0x7f521406bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 shutdown_connections 2024-09-21T05:46:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:01.537+0000 7f52198fb640 1 -- 172.21.15.28:0/1621754639 wait complete. 2024-09-21T05:46:02.351 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T05:46:02.351 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T05:46:02.360 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:46:02.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-21T05:46:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:02.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:02 smithi028 ceph-mon[31227]: pgmap v295: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:02.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:02 smithi181 ceph-mon[34819]: pgmap v295: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:02 smithi129 ceph-mon[34885]: pgmap v295: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 -- 172.21.15.28:0/3158065322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6381049b0 msgr2=0x7fa638104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 --2- 172.21.15.28:0/3158065322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6381049b0 0x7fa638104dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa628009900 tx=0x7fa62802a410 comp rx=0 tx=0).stop 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 -- 172.21.15.28:0/3158065322 shutdown_connections 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 --2- 172.21.15.28:0/3158065322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa63810de10 0x7fa638110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 --2- 172.21.15.28:0/3158065322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa638105300 0x7fa63810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 --2- 172.21.15.28:0/3158065322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6381049b0 0x7fa638104dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 -- 172.21.15.28:0/3158065322 >> 172.21.15.28:0/3158065322 conn(0x7fa638079730 msgr2=0x7fa638079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 -- 172.21.15.28:0/3158065322 shutdown_connections 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.064+0000 7fa63eec6640 1 -- 172.21.15.28:0/3158065322 wait complete. 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.065+0000 7fa63eec6640 1 Processor -- start 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.065+0000 7fa63eec6640 1 -- start start 2024-09-21T05:46:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6381049b0 0x7fa63819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa63810de10 0x7fa63819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa638113a90 con 0x7fa638105300 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa638113910 con 0x7fa6381049b0 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa638113c10 con 0x7fa63810de10 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55420/0 (socket says 172.21.15.28:55420) 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 -- 172.21.15.28:0/1832950532 learned_addr learned my addr 172.21.15.28:0/1832950532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63cc62640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6381049b0 0x7fa63819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63d463640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa63810de10 0x7fa63819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 -- 172.21.15.28:0/1832950532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa63810de10 msgr2=0x7fa63819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa63810de10 0x7fa63819fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 -- 172.21.15.28:0/1832950532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6381049b0 msgr2=0x7fa63819a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6381049b0 0x7fa63819a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 -- 172.21.15.28:0/1832950532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6381a04d0 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa637fff640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e9c0 tx=0x7fa62c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00ccc0 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62c0054d0 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.066+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6381a07c0 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.067+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c010430 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.067+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6381a6f20 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.068+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa638104dc0 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.069+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fa62c005020 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.070+0000 7fa635ffb640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 0x7fa6200b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.070+0000 7fa63cc62640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 0x7fa6200b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.070+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa62c0e7940 con 0x7fa638105300 2024-09-21T05:46:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.071+0000 7fa63cc62640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 0x7fa6200b8b80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa628031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.075+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa62c0adf10 con 0x7fa638105300 2024-09-21T05:46:03.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.286+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fa638063800 con 0x7fa6200b6670 2024-09-21T05:46:03.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.287+0000 7fa635ffb640 1 -- 172.21.15.28:0/1832950532 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fa638063800 con 0x7fa6200b6670 2024-09-21T05:46:03.290 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-09-21T05:46:03.290 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-09-21T05:46:03.290 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 msgr2=0x7fa6200b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 0x7fa6200b8b80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa628031400 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 msgr2=0x7fa63819ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e9c0 tx=0x7fa62c00ee80 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 shutdown_connections 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa63810de10 0x7fa63819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6200b6670 0x7fa6200b8b80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa638105300 0x7fa63819ab00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 --2- 172.21.15.28:0/1832950532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6381049b0 0x7fa63819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.289+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 >> 172.21.15.28:0/1832950532 conn(0x7fa638079730 msgr2=0x7fa6380ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.290+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 shutdown_connections 2024-09-21T05:46:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:03.290+0000 7fa63eec6640 1 -- 172.21.15.28:0/1832950532 wait complete. 2024-09-21T05:46:03.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-21T05:46:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:04.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:04 smithi028 ceph-mon[31227]: pgmap v296: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:04.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:04 smithi028 ceph-mon[31227]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2504778635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c01083a0 msgr2=0x7fc2c01087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2504778635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c01083a0 0x7fc2c01087b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009900 tx=0x7fc2b402a410 comp rx=0 tx=0).stop 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2504778635 shutdown_connections 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2504778635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c0108cf0 0x7fc2c010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2504778635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c01083a0 0x7fc2c01087b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2504778635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 0x7fc2c0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2504778635 >> 172.21.15.28:0/2504778635 conn(0x7fc2c0079730 msgr2=0x7fc2c0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2504778635 shutdown_connections 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.683+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2504778635 wait complete. 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 Processor -- start 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 -- start start 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 0x7fc2c019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c01083a0 0x7fc2c019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c0112960 con 0x7fc2c0108cf0 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01127e0 con 0x7fc2c01083a0 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2c58ab640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c0112ae0 con 0x7fc2c0102a60 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55446/0 (socket says 172.21.15.28:55446) 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.684+0000 7fc2bf7fe640 1 -- 172.21.15.28:0/2180125156 learned_addr learned my addr 172.21.15.28:0/2180125156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2beffd640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 0x7fc2c019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2be7fc640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c01083a0 0x7fc2c019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 -- 172.21.15.28:0/2180125156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 msgr2=0x7fc2c019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 0x7fc2c019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 -- 172.21.15.28:0/2180125156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c01083a0 msgr2=0x7fc2c019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c01083a0 0x7fc2c019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 -- 172.21.15.28:0/2180125156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c01a0330 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2bf7fe640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800cce0 tx=0x7fc2b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800f450 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c01a0620 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b800faa0 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8018b70 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.685+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c01a0ae0 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.686+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c0102e70 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.688+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fc2b8018d10 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.689+0000 7fc2c48a9640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 0x7fc2a80b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.689+0000 7fc2beffd640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 0x7fc2a80b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.689+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc2b80e7ad0 con 0x7fc2c0108cf0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.690+0000 7fc2beffd640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 0x7fc2a80b8b30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000bbe0 tx=0x7fc2b000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.693+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc2b8014030 con 0x7fc2c0108cf0 2024-09-21T05:46:04.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:04 smithi181 ceph-mon[34819]: pgmap v296: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:04.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:04 smithi181 ceph-mon[34819]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:04 smithi129 ceph-mon[34885]: pgmap v296: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:04 smithi129 ceph-mon[34885]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.894+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fc2c0063800 con 0x7fc2a80b6620 2024-09-21T05:46:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.907+0000 7fc2c48a9640 1 -- 172.21.15.28:0/2180125156 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fc2c0063800 con 0x7fc2a80b6620 2024-09-21T05:46:04.910 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (98s) 20s ago 98s 85.2M - 19.3.0-5120-g18432218 ddce307e1f32 284974c41521 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi129 running (96s) 29s ago 95s 87.8M - 19.3.0-5120-g18432218 ddce307e1f32 360c65f6c9a9 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi181 running (109s) 29s ago 108s 87.9M - 19.3.0-5120-g18432218 ddce307e1f32 602d09a33006 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (115s) 20s ago 115s 82.2M - 19.3.0-5120-g18432218 ddce307e1f32 5fa80b0b69ff 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi129 running (113s) 29s ago 113s 84.9M - 19.3.0-5120-g18432218 ddce307e1f32 7ddfa32c3fe6 2024-09-21T05:46:04.911 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi181 running (111s) 29s ago 111s 86.4M - 19.3.0-5120-g18432218 ddce307e1f32 22dda338f927 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (101s) 20s ago 100s 84.1M - 19.3.0-5120-g18432218 ddce307e1f32 2596bd3e580d 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi129 running (106s) 29s ago 106s 87.7M - 19.3.0-5120-g18432218 ddce307e1f32 70ca59e840c9 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi181 running (104s) 29s ago 103s 94.1M - 19.3.0-5120-g18432218 ddce307e1f32 4ef35a78ca72 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (119s) 20s ago 118s 83.3M - 19.3.0-5120-g18432218 ddce307e1f32 61a456e61e81 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi129 running (2m) 29s ago 2m 84.4M - 19.3.0-5120-g18432218 ddce307e1f32 e80cfed98299 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi181 running (2m) 29s ago 2m 84.6M - 19.3.0-5120-g18432218 ddce307e1f32 17bfa00145c3 2024-09-21T05:46:04.912 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (9m) 20s ago 9m 588M - 19.3.0-5120-g18432218 ddce307e1f32 4efa3b081274 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi129 *:8443,8765 running (7m) 29s ago 7m 525M - 19.3.0-5120-g18432218 ddce307e1f32 83a052978e29 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi181 *:8443,8765 running (7m) 29s ago 7m 525M - 19.3.0-5120-g18432218 ddce307e1f32 3dfa4497666f 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (9m) 20s ago 9m 119M 2048M 19.3.0-5120-g18432218 ddce307e1f32 a73d3a9255f0 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi129 running (8m) 29s ago 8m 110M 2048M 19.3.0-5120-g18432218 ddce307e1f32 d1186cbf6d5b 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi181 running (8m) 29s ago 8m 110M 2048M 19.3.0-5120-g18432218 ddce307e1f32 509a1ea6af6e 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 20s ago 6m 116M 4273M 19.3.0-5120-g18432218 ddce307e1f32 e22c7ed3a590 2024-09-21T05:46:04.913 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi129 running (6m) 29s ago 6m 121M 4272M 19.3.0-5120-g18432218 ddce307e1f32 54bcc27ac795 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi181 running (6m) 29s ago 6m 125M 4272M 19.3.0-5120-g18432218 ddce307e1f32 55ead999486c 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (5m) 20s ago 5m 113M 4273M 19.3.0-5120-g18432218 ddce307e1f32 0295a955ff3e 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi129 running (5m) 29s ago 5m 122M 4272M 19.3.0-5120-g18432218 ddce307e1f32 8984b4e9355c 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi181 running (4m) 29s ago 4m 113M 4272M 19.3.0-5120-g18432218 ddce307e1f32 269c4aa36ace 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (4m) 20s ago 4m 113M 4273M 19.3.0-5120-g18432218 ddce307e1f32 e7a04689c453 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi129 running (3m) 29s ago 3m 121M 4272M 19.3.0-5120-g18432218 ddce307e1f32 5129eea019cd 2024-09-21T05:46:04.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi181 running (3m) 29s ago 3m 123M 4272M 19.3.0-5120-g18432218 ddce307e1f32 53df391d8b5e 2024-09-21T05:46:04.915 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 20s ago 3m 118M 4273M 19.3.0-5120-g18432218 ddce307e1f32 ea8413345b44 2024-09-21T05:46:04.915 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi129 running (2m) 29s ago 2m 122M 4272M 19.3.0-5120-g18432218 ddce307e1f32 26b55f2e278c 2024-09-21T05:46:04.915 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi181 running (2m) 29s ago 2m 119M 4272M 19.3.0-5120-g18432218 ddce307e1f32 ef3c7559dbff 2024-09-21T05:46:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 msgr2=0x7fc2a80b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 0x7fc2a80b8b30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000bbe0 tx=0x7fc2b000b390 comp rx=0 tx=0).stop 2024-09-21T05:46:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 msgr2=0x7fc2c019fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800cce0 tx=0x7fc2b800a620 comp rx=0 tx=0).stop 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 shutdown_connections 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2a80b6620 0x7fc2a80b8b30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c0108cf0 0x7fc2c019fcc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2c01083a0 0x7fc2c019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 --2- 172.21.15.28:0/2180125156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2c0102a60 0x7fc2c019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 >> 172.21.15.28:0/2180125156 conn(0x7fc2c0079730 msgr2=0x7fc2c006b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 shutdown_connections 2024-09-21T05:46:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:04.910+0000 7fc2c58ab640 1 -- 172.21.15.28:0/2180125156 wait complete. 2024-09-21T05:46:05.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-21T05:46:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 -- 172.21.15.28:0/4275837665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a001049f0 msgr2=0x7f8a00104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/4275837665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a001049f0 0x7f8a00104e00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009900 tx=0x7f89f002a410 comp rx=0 tx=0).stop 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 -- 172.21.15.28:0/4275837665 shutdown_connections 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/4275837665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a0010de50 0x7f8a00110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/4275837665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a00105340 0x7f8a0010d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/4275837665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a001049f0 0x7f8a00104e00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 -- 172.21.15.28:0/4275837665 >> 172.21.15.28:0/4275837665 conn(0x7f8a000797e0 msgr2=0x7f8a00079bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 -- 172.21.15.28:0/4275837665 shutdown_connections 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.399+0000 7f8a04dd5640 1 -- 172.21.15.28:0/4275837665 wait complete. 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 Processor -- start 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 -- start start 2024-09-21T05:46:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a001049f0 0x7f8a0019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0010de50 0x7f8a0019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00113ab0 con 0x7f8a00105340 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00113930 con 0x7f8a0010de50 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f8a04dd5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00113c30 con 0x7f8a001049f0 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45432/0 (socket says 172.21.15.28:45432) 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 -- 172.21.15.28:0/1059916892 learned_addr learned my addr 172.21.15.28:0/1059916892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fe59c640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a001049f0 0x7f8a0019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fed9d640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0010de50 0x7f8a0019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 -- 172.21.15.28:0/1059916892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a001049f0 msgr2=0x7f8a0019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a001049f0 0x7f8a0019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 -- 172.21.15.28:0/1059916892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0010de50 msgr2=0x7f8a0019fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0010de50 0x7f8a0019fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.400+0000 7f89fdd9b640 1 -- 172.21.15.28:0/1059916892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a001a0490 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f89fdd9b640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89f400d950 tx=0x7f89f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4017070 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89f4009c30 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a001a0780 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f40047a0 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.401+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001a6ee0 con 0x7f8a00105340 2024-09-21T05:46:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.402+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00104e00 con 0x7f8a00105340 2024-09-21T05:46:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.404+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f89f4004020 con 0x7f8a00105340 2024-09-21T05:46:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.405+0000 7f89ef7fe640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 0x7f89e80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.405+0000 7f89fe59c640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 0x7f89e80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.405+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f89f40e72d0 con 0x7f8a00105340 2024-09-21T05:46:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.405+0000 7f89fe59c640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 0x7f89e80b8b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:06.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.409+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f89f4020020 con 0x7f8a00105340 2024-09-21T05:46:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:06 smithi028 ceph-mon[31227]: pgmap v297: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:06 smithi028 ceph-mon[31227]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.605+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8a00108080 con 0x7f89e80b6670 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.611+0000 7f89ef7fe640 1 -- 172.21.15.28:0/1059916892 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8a00108080 con 0x7f89e80b6670 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 31s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 31s ago 7m smithi028=x;smithi129=y;smithi181=z;count:3 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 31s ago 8m smithi028:172.21.15.28=a;smithi129:172.21.15.129=b;smithi181:172.21.15.181=c;count:3 2024-09-21T05:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 31s ago - 2024-09-21T05:46:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 msgr2=0x7f89e80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 0x7f89e80b8b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0031400 comp rx=0 tx=0).stop 2024-09-21T05:46:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 msgr2=0x7f8a0019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89f400d950 tx=0x7f89f400de10 comp rx=0 tx=0).stop 2024-09-21T05:46:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 shutdown_connections 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0010de50 0x7f8a0019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89e80b6670 0x7f89e80b8b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00105340 0x7f8a0019aac0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 --2- 172.21.15.28:0/1059916892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a001049f0 0x7f8a0019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 >> 172.21.15.28:0/1059916892 conn(0x7f8a000797e0 msgr2=0x7f8a00107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 shutdown_connections 2024-09-21T05:46:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:06.613+0000 7f8a04dd5640 1 -- 172.21.15.28:0/1059916892 wait complete. 2024-09-21T05:46:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:06 smithi181 ceph-mon[34819]: pgmap v297: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:06 smithi181 ceph-mon[34819]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:06 smithi129 ceph-mon[34885]: pgmap v297: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:06 smithi129 ceph-mon[34885]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:07.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-21T05:46:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 -- 172.21.15.28:0/2229087818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd950109df0 msgr2=0x7fd950110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 --2- 172.21.15.28:0/2229087818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd950109df0 0x7fd950110620 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd948009ca0 tx=0x7fd94802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 -- 172.21.15.28:0/2229087818 shutdown_connections 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 --2- 172.21.15.28:0/2229087818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd950109df0 0x7fd950110620 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 --2- 172.21.15.28:0/2229087818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9501094a0 0x7fd9501098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 --2- 172.21.15.28:0/2229087818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950103b60 0x7fd950103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 -- 172.21.15.28:0/2229087818 >> 172.21.15.28:0/2229087818 conn(0x7fd95007b850 msgr2=0x7fd95007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 -- 172.21.15.28:0/2229087818 shutdown_connections 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.030+0000 7fd95500f640 1 -- 172.21.15.28:0/2229087818 wait complete. 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.031+0000 7fd95500f640 1 Processor -- start 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 -- start start 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 0x7fd95019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950109df0 0x7fd95019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd950113b10 con 0x7fd9501094a0 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd950113990 con 0x7fd950103b60 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd95500f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd950113c90 con 0x7fd950109df0 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45452/0 (socket says 172.21.15.28:45452) 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94e59c640 1 -- 172.21.15.28:0/783531680 learned_addr learned my addr 172.21.15.28:0/783531680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94ed9d640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 0x7fd95019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94f59e640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950109df0 0x7fd95019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94e59c640 1 -- 172.21.15.28:0/783531680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950109df0 msgr2=0x7fd95019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.032+0000 7fd94e59c640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950109df0 0x7fd95019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd94e59c640 1 -- 172.21.15.28:0/783531680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 msgr2=0x7fd95019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd94e59c640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 0x7fd95019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd94e59c640 1 -- 172.21.15.28:0/783531680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9501a0330 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd94ed9d640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 0x7fd95019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd94e59c640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c8c0 tx=0x7fd94400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd944007bb0 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9501a0620 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd944007d50 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9440187c0 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.033+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9501a6e40 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.034+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd950103f70 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.036+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fd94400fa20 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.036+0000 7fd937fff640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 0x7fd9380b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.037+0000 7fd94ed9d640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 0x7fd9380b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.037+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd9440e7a30 con 0x7fd9501094a0 2024-09-21T05:46:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.037+0000 7fd94ed9d640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 0x7fd9380b8b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd940009930 tx=0x7fd940008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.040+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd944014030 con 0x7fd9501094a0 2024-09-21T05:46:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.235+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fd950063800 con 0x7fd9380b6620 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.237+0000 7fd937fff640 1 -- 172.21.15.28:0/783531680 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fd950063800 con 0x7fd9380b6620 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi129 172.21.15.129 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi181 172.21.15.181 2024-09-21T05:46:08.240 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-09-21T05:46:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 msgr2=0x7fd9380b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 0x7fd9380b8b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd940009930 tx=0x7fd940008040 comp rx=0 tx=0).stop 2024-09-21T05:46:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 msgr2=0x7fd95019aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c8c0 tx=0x7fd94400cd80 comp rx=0 tx=0).stop 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 shutdown_connections 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd950109df0 0x7fd95019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd9380b6620 0x7fd9380b8b30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9501094a0 0x7fd95019aa20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 --2- 172.21.15.28:0/783531680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd950103b60 0x7fd95019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 >> 172.21.15.28:0/783531680 conn(0x7fd95007b850 msgr2=0x7fd9501074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 shutdown_connections 2024-09-21T05:46:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:08.239+0000 7fd95500f640 1 -- 172.21.15.28:0/783531680 wait complete. 2024-09-21T05:46:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:08 smithi028 ceph-mon[31227]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:08 smithi028 ceph-mon[31227]: pgmap v298: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:08 smithi181 ceph-mon[34819]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:08 smithi181 ceph-mon[34819]: pgmap v298: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:08 smithi129 ceph-mon[34885]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:08 smithi129 ceph-mon[34885]: pgmap v298: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:46:08.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-21T05:46:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:09 smithi028 ceph-mon[31227]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.629+0000 7fdaa55ca640 1 -- 172.21.15.28:0/4099032859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 msgr2=0x7fdaa0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/4099032859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa0104c90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fda90009900 tx=0x7fda9002a410 comp rx=0 tx=0).stop 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 -- 172.21.15.28:0/4099032859 shutdown_connections 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/4099032859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa010de30 0x7fdaa0110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/4099032859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa01052b0 0x7fdaa010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/4099032859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa0104c90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.630+0000 7fdaa55ca640 1 -- 172.21.15.28:0/4099032859 >> 172.21.15.28:0/4099032859 conn(0x7fdaa0079750 msgr2=0x7fdaa0079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.631+0000 7fdaa55ca640 1 -- 172.21.15.28:0/4099032859 shutdown_connections 2024-09-21T05:46:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.631+0000 7fdaa55ca640 1 -- 172.21.15.28:0/4099032859 wait complete. 2024-09-21T05:46:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.631+0000 7fdaa55ca640 1 Processor -- start 2024-09-21T05:46:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 -- start start 2024-09-21T05:46:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 0x7fdaa019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa010de30 0x7fdaa019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa0113ad0 con 0x7fdaa0104880 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa0113950 con 0x7fdaa01052b0 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fdaa55ca640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa0113c50 con 0x7fdaa010de30 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45470/0 (socket says 172.21.15.28:45470) 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 -- 172.21.15.28:0/3923151911 learned_addr learned my addr 172.21.15.28:0/3923151911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9f7fe640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa010de30 0x7fdaa019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9e7fc640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 0x7fdaa019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 -- 172.21.15.28:0/3923151911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa010de30 msgr2=0x7fdaa019fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa010de30 0x7fdaa019fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 -- 172.21.15.28:0/3923151911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 msgr2=0x7fdaa019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 0x7fdaa019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9effd640 1 -- 172.21.15.28:0/3923151911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa01a0410 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.632+0000 7fda9e7fc640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 0x7fdaa019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fda9effd640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fda90005940 tx=0x7fda90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9003f030 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda90003aa0 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa01a06a0 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90035070 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.633+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa01a6e60 con 0x7fdaa0104880 2024-09-21T05:46:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.634+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa006c1e0 con 0x7fdaa0104880 2024-09-21T05:46:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.635+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fda90042030 con 0x7fdaa0104880 2024-09-21T05:46:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.636+0000 7fda87fff640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 0x7fda880b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.636+0000 7fda9e7fc640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 0x7fda880b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.636+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fda90107f90 con 0x7fdaa0104880 2024-09-21T05:46:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.637+0000 7fda9e7fc640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 0x7fda880b8b80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdaa010fbc0 tx=0x7fda94005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.640+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fda900ce5c0 con 0x7fdaa0104880 2024-09-21T05:46:09.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:09 smithi181 ceph-mon[34819]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.835+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fdaa0101d50 con 0x7fda880b6670 2024-09-21T05:46:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.837+0000 7fda87fff640 1 -- 172.21.15.28:0/3923151911 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fdaa0101d50 con 0x7fda880b6670 2024-09-21T05:46:09.840 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-21T05:46:09.840 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-21T05:46:09.840 INFO:teuthology.orchestra.run.smithi028.stdout:smithi129 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300HY400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-21T05:46:09.840 INFO:teuthology.orchestra.run.smithi028.stdout:smithi181 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000E400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 msgr2=0x7fda880b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 0x7fda880b8b80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdaa010fbc0 tx=0x7fda94005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 msgr2=0x7fdaa019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fda90005940 tx=0x7fda90004270 comp rx=0 tx=0).stop 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 shutdown_connections 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaa010de30 0x7fdaa019fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa01052b0 0x7fdaa019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fda880b6670 0x7fda880b8b80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 --2- 172.21.15.28:0/3923151911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa0104880 0x7fdaa019a4a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 >> 172.21.15.28:0/3923151911 conn(0x7fdaa0079750 msgr2=0x7fdaa0100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 shutdown_connections 2024-09-21T05:46:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:09.839+0000 7fdaa55ca640 1 -- 172.21.15.28:0/3923151911 wait complete. 2024-09-21T05:46:09.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:09 smithi129 ceph-mon[34885]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:10.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:10 smithi028 ceph-mon[31227]: pgmap v299: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:10.555 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T05:46:10.564 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:46:10.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-21T05:46:10.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:10 smithi181 ceph-mon[34819]: pgmap v299: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:10.831 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:10.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:10 smithi129 ceph-mon[34885]: pgmap v299: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 -- 172.21.15.28:0/3203450435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 msgr2=0x7f684c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 --2- 172.21.15.28:0/3203450435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c1087b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6834009900 tx=0x7f683402a410 comp rx=0 tx=0).stop 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 -- 172.21.15.28:0/3203450435 shutdown_connections 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 --2- 172.21.15.28:0/3203450435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c108cf0 0x7f684c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 --2- 172.21.15.28:0/3203450435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c1087b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 --2- 172.21.15.28:0/3203450435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c102a60 0x7f684c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.279+0000 7f6850a42640 1 -- 172.21.15.28:0/3203450435 >> 172.21.15.28:0/3203450435 conn(0x7f684c079730 msgr2=0x7f684c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.280+0000 7f6850a42640 1 -- 172.21.15.28:0/3203450435 shutdown_connections 2024-09-21T05:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.280+0000 7f6850a42640 1 -- 172.21.15.28:0/3203450435 wait complete. 2024-09-21T05:46:11.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.280+0000 7f6850a42640 1 Processor -- start 2024-09-21T05:46:11.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.280+0000 7f6850a42640 1 -- start start 2024-09-21T05:46:11.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c102a60 0x7f684c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:11.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:11.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 0x7f684c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f684c112960 con 0x7f684c1083a0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1127e0 con 0x7f684c108cf0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6850a42640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f684c112ae0 con 0x7f684c102a60 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45488/0 (socket says 172.21.15.28:45488) 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6843fff640 1 -- 172.21.15.28:0/2111020322 learned_addr learned my addr 172.21.15.28:0/2111020322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f684a59c640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c102a60 0x7f684c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f684ad9d640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 0x7f684c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6843fff640 1 -- 172.21.15.28:0/2111020322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c102a60 msgr2=0x7f684c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.281+0000 7f6843fff640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c102a60 0x7f684c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6843fff640 1 -- 172.21.15.28:0/2111020322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 msgr2=0x7f684c19be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6843fff640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 0x7f684c19be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6843fff640 1 -- 172.21.15.28:0/2111020322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684c1a6620 con 0x7f684c1083a0 2024-09-21T05:46:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f684ad9d640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 0x7f684c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6843fff640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6834032fd0 tx=0x7f6834004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683403f030 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6834004490 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684c1a68b0 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834035070 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.282+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684c1a6d90 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.283+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684c102e70 con 0x7f684c1083a0 2024-09-21T05:46:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.285+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f6834042030 con 0x7f684c1083a0 2024-09-21T05:46:11.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.285+0000 7f68437fe640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 0x7f68380b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:11.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.286+0000 7f684a59c640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 0x7f68380b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:11.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.286+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6834107ad0 con 0x7f684c1083a0 2024-09-21T05:46:11.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.286+0000 7f684a59c640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 0x7f68380b8b30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f683c0058c0 tx=0x7f683c00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.290+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f68340ce100 con 0x7f684c1083a0 2024-09-21T05:46:11.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.514+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f684c10ab40 con 0x7f684c1083a0 2024-09-21T05:46:11.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:11 smithi028 ceph-mon[31227]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.515+0000 7f68437fe640 1 -- 172.21.15.28:0/2111020322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3137 (secure 0 0 0) 0x7f68340d24e0 con 0x7f684c1083a0 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stdout:e26 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stdout:btime 2024-09-21T05:44:52:047664+0000 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T05:46:11.517 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 26 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-21T05:44:39.761973+0000 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-21T05:44:52.047653+0000 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-09-21T05:46:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 5 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2,3,4 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24688,1=24680,2=24673,3=24664,4=24653} 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3] 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-09-21T05:46:11.519 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24688 members: 24653,24664,24673,24680,24688 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{0:24688} state up:active seq 6 addr [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{0:24638} state up:standby-replay seq 1 addr [v2:172.21.15.129:6834/2842004697,v1:172.21.15.129:6835/2842004697] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{1:24680} state up:active seq 6 addr [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{1:24622} state up:standby-replay seq 1 addr [v2:172.21.15.181:6832/1938433457,v1:172.21.15.181:6833/1938433457] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24673} state up:active seq 10 addr [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{2:24614} state up:standby-replay seq 1 addr [v2:172.21.15.129:6832/2193748712,v1:172.21.15.129:6833/2193748712] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{3:24664} state up:active seq 11 addr [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{3:14856} state up:standby-replay seq 1 addr [v2:172.21.15.28:6840/2765835727,v1:172.21.15.28:6841/2765835727] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{4:24653} state up:active seq 10 addr [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{4:14850} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/2924711217,v1:172.21.15.28:6839/2924711217] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14784} state up:standby seq 1 addr [v2:172.21.15.28:6834/2759619114,v1:172.21.15.28:6835/2759619114] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14796} state up:standby seq 1 addr [v2:172.21.15.28:6836/783262834,v1:172.21.15.28:6837/783262834] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T05:46:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 msgr2=0x7f68380b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 0x7f68380b8b30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f683c0058c0 tx=0x7f683c00a2d0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 msgr2=0x7f684c19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6834032fd0 tx=0x7f6834004270 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 shutdown_connections 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c108cf0 0x7f684c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f68380b6620 0x7f68380b8b30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c1083a0 0x7f684c19b8d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 --2- 172.21.15.28:0/2111020322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c102a60 0x7f684c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 >> 172.21.15.28:0/2111020322 conn(0x7f684c079730 msgr2=0x7f684c104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 shutdown_connections 2024-09-21T05:46:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:11.517+0000 7f6850a42640 1 -- 172.21.15.28:0/2111020322 wait complete. 2024-09-21T05:46:11.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:11 smithi181 ceph-mon[34819]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:11.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:11 smithi129 ceph-mon[34885]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:12.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42500114-77db-11ef-baf5-efdc52797490 -- bash -c 'ceph osd dump' 2024-09-21T05:46:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:12 smithi028 ceph-mon[31227]: pgmap v300: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:12 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2111020322' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T05:46:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.786+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2914638428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a41046b0 msgr2=0x7fb4a4104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.786+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2914638428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a41046b0 0x7fb4a4104ac0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009900 tx=0x7fb4a002a410 comp rx=0 tx=0).stop 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2914638428 shutdown_connections 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2914638428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41028e0 0x7fb4a4112570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2914638428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a4101dc0 0x7fb4a41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2914638428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a41046b0 0x7fb4a4104ac0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2914638428 >> 172.21.15.28:0/2914638428 conn(0x7fb4a4079f90 msgr2=0x7fb4a407a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2914638428 shutdown_connections 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2914638428 wait complete. 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.787+0000 7fb4ab8c0640 1 Processor -- start 2024-09-21T05:46:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 -- start start 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a41028e0 0x7fb4a419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 0x7fb4a41a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4115d00 con 0x7fb4a4101dc0 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4115b80 con 0x7fb4a41028e0 2024-09-21T05:46:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4ab8c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4115e80 con 0x7fb4a41046b0 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 -- 172.21.15.28:0/2489592004 learned_addr learned my addr 172.21.15.28:0/2489592004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a8e5b640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a41028e0 0x7fb4a419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a9e5d640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 0x7fb4a41a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 -- 172.21.15.28:0/2489592004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 msgr2=0x7fb4a41a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 0x7fb4a41a1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 -- 172.21.15.28:0/2489592004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a41028e0 msgr2=0x7fb4a419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a41028e0 0x7fb4a419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a965c640 1 -- 172.21.15.28:0/2489592004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a41a26e0 con 0x7fb4a4101dc0 2024-09-21T05:46:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.788+0000 7fb4a9e5d640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 0x7fb4a41a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb4a965c640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005940 tx=0x7fb4a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a003f030 con 0x7fb4a4101dc0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4a0003aa0 con 0x7fb4a4101dc0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a41a2970 con 0x7fb4a4101dc0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0035070 con 0x7fb4a4101dc0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.789+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a41a9130 con 0x7fb4a4101dc0 2024-09-21T05:46:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.790+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a41021d0 con 0x7fb4a4101dc0 2024-09-21T05:46:12.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:12 smithi181 ceph-mon[34819]: pgmap v300: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:12.795 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:12 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2111020322' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T05:46:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.792+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7fb4a0042030 con 0x7fb4a4101dc0 2024-09-21T05:46:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.792+0000 7fb49a7fc640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 0x7fb4900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.792+0000 7fb4a8e5b640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 0x7fb4900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.792+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb4a0107f70 con 0x7fb4a4101dc0 2024-09-21T05:46:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.793+0000 7fb4a8e5b640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 0x7fb4900b8b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb4a407a8b0 tx=0x7fb494005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.796+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb4a00ce5a0 con 0x7fb4a4101dc0 2024-09-21T05:46:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:12 smithi129 ceph-mon[34885]: pgmap v300: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:12 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2111020322' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T05:46:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.986+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fb4a41a9300 con 0x7fb4a4101dc0 2024-09-21T05:46:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.987+0000 7fb49a7fc640 1 -- 172.21.15.28:0/2489592004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v73) ==== 56+0+5717 (secure 0 0 0) 0x7fb4a00d2980 con 0x7fb4a4101dc0 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 73 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-21T05:36:12.179132+0000 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-21T05:45:14.248651+0000 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 26 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-09-21T05:46:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 9 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/3566179718,v1:172.21.15.28:6803/3566179718] [v2:172.21.15.28:6804/3566179718,v1:172.21.15.28:6805/3566179718] exists,up aebf5461-2659-41f2-8c87-f7d8bce2cc0d 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 14 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6800/1623835479,v1:172.21.15.129:6801/1623835479] [v2:172.21.15.129:6802/1623835479,v1:172.21.15.129:6803/1623835479] exists,up ced7a484-b9ec-49ce-825a-16be7ba41ec2 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/3260464300,v1:172.21.15.181:6801/3260464300] [v2:172.21.15.181:6802/3260464300,v1:172.21.15.181:6803/3260464300] exists,up 0efbfaff-07a3-4354-b453-5b1ee7fea224 2024-09-21T05:46:12.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 24 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/2300237806,v1:172.21.15.28:6811/2300237806] [v2:172.21.15.28:6812/2300237806,v1:172.21.15.28:6813/2300237806] exists,up 7e76bace-e04c-43a5-95d6-753d39f5adb3 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 30 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6808/645644998,v1:172.21.15.129:6809/645644998] [v2:172.21.15.129:6810/645644998,v1:172.21.15.129:6811/645644998] exists,up e34f6ec9-ba14-4098-af52-2c147d72cba5 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 34 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/1579505960,v1:172.21.15.181:6809/1579505960] [v2:172.21.15.181:6810/1579505960,v1:172.21.15.181:6811/1579505960] exists,up 76455120-2b9e-4fe9-9ae7-3f85cf2a54c9 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 39 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/88277262,v1:172.21.15.28:6819/88277262] [v2:172.21.15.28:6820/88277262,v1:172.21.15.28:6821/88277262] exists,up 7a460509-dbe4-4d67-a88a-d45cf3c21227 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 44 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6816/317230964,v1:172.21.15.129:6817/317230964] [v2:172.21.15.129:6818/317230964,v1:172.21.15.129:6819/317230964] exists,up 9ed1f260-f7ce-4489-80a1-043099519502 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 50 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/3198301680,v1:172.21.15.181:6817/3198301680] [v2:172.21.15.181:6818/3198301680,v1:172.21.15.181:6819/3198301680] exists,up 2a7581c5-4c13-4d96-82a4-c910a21c857c 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 55 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/2894928742,v1:172.21.15.28:6827/2894928742] [v2:172.21.15.28:6828/2894928742,v1:172.21.15.28:6829/2894928742] exists,up 0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6824/3703525783,v1:172.21.15.129:6825/3703525783] [v2:172.21.15.129:6826/3703525783,v1:172.21.15.129:6827/3703525783] exists,up 8d6e3073-eab4-4ac9-a926-c42fa4e501d0 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 65 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/3814712780,v1:172.21.15.181:6825/3814712780] [v2:172.21.15.181:6826/3814712780,v1:172.21.15.181:6827/3814712780] exists,up f7d4e141-618d-41ae-a3e8-4120e69bc428 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.5 [8,11] 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.16 [2,4] 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.18 [5,11] 2024-09-21T05:46:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1a [8,1] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1b [2,4] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.7 [3,9] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.e [4,8] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1d [5,9] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2c [4,9] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3a [4,6] 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3351178737 expires 2024-09-22T05:38:12.622156+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1190896102 expires 2024-09-22T05:38:12.622156+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2589278469 expires 2024-09-22T05:36:43.853731+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2066463820 expires 2024-09-22T05:36:43.853731+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3235284089 expires 2024-09-22T05:36:43.853731+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/689076330 expires 2024-09-22T05:37:11.040387+0000 2024-09-21T05:46:12.992 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3969089930 expires 2024-09-22T05:37:11.040387+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1887508254 expires 2024-09-22T05:37:11.040387+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1617434937 expires 2024-09-22T05:38:12.622156+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2299789749 expires 2024-09-22T05:36:43.853731+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/689076330 expires 2024-09-22T05:37:11.040387+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4032693217 expires 2024-09-22T05:37:11.040387+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3235284089 expires 2024-09-22T05:36:43.853731+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2040548036 expires 2024-09-22T05:38:12.622156+0000 2024-09-21T05:46:12.993 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2040548036 expires 2024-09-22T05:38:12.622156+0000 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 msgr2=0x7fb4900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 0x7fb4900b8b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb4a407a8b0 tx=0x7fb494005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 msgr2=0x7fb4a419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005940 tx=0x7fb4a0004270 comp rx=0 tx=0).stop 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 shutdown_connections 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4a41046b0 0x7fb4a41a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a41028e0 0x7fb4a419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4900b65f0 0x7fb4900b8b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 --2- 172.21.15.28:0/2489592004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4a4101dc0 0x7fb4a419c770 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.990+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 >> 172.21.15.28:0/2489592004 conn(0x7fb4a4079f90 msgr2=0x7fb4a4100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.991+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 shutdown_connections 2024-09-21T05:46:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:12.991+0000 7fb4ab8c0640 1 -- 172.21.15.28:0/2489592004 wait complete. 2024-09-21T05:46:13.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:13 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2489592004' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T05:46:13.570 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-21T05:46:13.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:46:13.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:13 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2489592004' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T05:46:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:13.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:13 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2489592004' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 -- 172.21.15.28:0/3664717105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 msgr2=0x7f46dc1125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 --2- 172.21.15.28:0/3664717105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1125c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009ca0 tx=0x7f46d402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 -- 172.21.15.28:0/3664717105 shutdown_connections 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 --2- 172.21.15.28:0/3664717105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1125c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 --2- 172.21.15.28:0/3664717105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 0x7f46dc10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 --2- 172.21.15.28:0/3664717105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 0x7f46dc075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 -- 172.21.15.28:0/3664717105 >> 172.21.15.28:0/3664717105 conn(0x7f46dc07b910 msgr2=0x7f46dc07bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 -- 172.21.15.28:0/3664717105 shutdown_connections 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.317+0000 7f46e255c640 1 -- 172.21.15.28:0/3664717105 wait complete. 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 Processor -- start 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 -- start start 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 0x7f46dc19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 0x7f46dc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc115cb0 con 0x7f46dc1101a0 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc115b30 con 0x7f46dc075730 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e255c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc115e30 con 0x7f46dc076080 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e0af9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e0af9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45530/0 (socket says 172.21.15.28:45530) 2024-09-21T05:46:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46e0af9640 1 -- 172.21.15.28:0/689602514 learned_addr learned my addr 172.21.15.28:0/689602514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.318+0000 7f46dbfff640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 0x7f46dc19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46db7fe640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 0x7f46dc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 -- 172.21.15.28:0/689602514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 msgr2=0x7f46dc19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 0x7f46dc19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 -- 172.21.15.28:0/689602514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 msgr2=0x7f46dc19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 0x7f46dc19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 -- 172.21.15.28:0/689602514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc1a2590 con 0x7f46dc1101a0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e0af9640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f46d40098d0 tx=0x7f46d40317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d4003b70 con 0x7f46dc1101a0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc1a2820 con 0x7f46dc1101a0 2024-09-21T05:46:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d4003d10 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d402fa20 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.319+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc1a8fe0 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.320+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46dc075b40 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.322+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 154580+0+0 (secure 0 0 0) 0x7f46d402fc40 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.322+0000 7f46d97fa640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 0x7f46c40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.323+0000 7f46dbfff640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 0x7f46c40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.323+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f46d41088e0 con 0x7f46dc1101a0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.323+0000 7f46dbfff640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 0x7f46c40b8b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009bd0 tx=0x7f46cc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.327+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46d40cef10 con 0x7f46dc1101a0 2024-09-21T05:46:14.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.556+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46dc1a91b0 con 0x7f46dc1101a0 2024-09-21T05:46:14.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:14 smithi028 ceph-mon[31227]: pgmap v301: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.557+0000 7f46d97fa640 1 -- 172.21.15.28:0/689602514 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7f46d40d32f0 con 0x7f46dc1101a0 2024-09-21T05:46:14.559 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:14.559 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:46:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 msgr2=0x7f46c40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 0x7f46c40b8b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009bd0 tx=0x7f46cc005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 msgr2=0x7f46dc1a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f46d40098d0 tx=0x7f46d40317b0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 shutdown_connections 2024-09-21T05:46:14.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46c40b65f0 0x7f46c40b8b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46dc1101a0 0x7f46dc1a1e60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46dc076080 0x7f46dc19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 --2- 172.21.15.28:0/689602514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46dc075730 0x7f46dc19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 >> 172.21.15.28:0/689602514 conn(0x7f46dc07b910 msgr2=0x7f46dc109c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 shutdown_connections 2024-09-21T05:46:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:14.559+0000 7f46e255c640 1 -- 172.21.15.28:0/689602514 wait complete. 2024-09-21T05:46:14.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:14 smithi181 ceph-mon[34819]: pgmap v301: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:14 smithi129 ceph-mon[34885]: pgmap v301: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T05:46:15.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:46:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:15.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:15 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/689602514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:15.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:15 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/689602514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:15.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:15 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/689602514' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 -- 172.21.15.28:0/2400353970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974108d20 msgr2=0x7f597410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 --2- 172.21.15.28:0/2400353970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974108d20 0x7f597410f550 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f596c009ca0 tx=0x7f596c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 -- 172.21.15.28:0/2400353970 shutdown_connections 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 --2- 172.21.15.28:0/2400353970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974108d20 0x7f597410f550 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 --2- 172.21.15.28:0/2400353970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59741083d0 0x7f59741087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 --2- 172.21.15.28:0/2400353970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f5974102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 -- 172.21.15.28:0/2400353970 >> 172.21.15.28:0/2400353970 conn(0x7f5974079760 msgr2=0x7f5974079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 -- 172.21.15.28:0/2400353970 shutdown_connections 2024-09-21T05:46:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 -- 172.21.15.28:0/2400353970 wait complete. 2024-09-21T05:46:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.918+0000 7f5978a8e640 1 Processor -- start 2024-09-21T05:46:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 -- start start 2024-09-21T05:46:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f597419b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5974108d20 0x7f597419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5974112990 con 0x7f59741083d0 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5974112810 con 0x7f5974102ae0 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5978a8e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5974112b10 con 0x7f5974108d20 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44224/0 (socket says 172.21.15.28:44224) 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 -- 172.21.15.28:0/3605329236 learned_addr learned my addr 172.21.15.28:0/3605329236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f597259c640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f597419b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f5972d9d640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5974108d20 0x7f597419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 -- 172.21.15.28:0/3605329236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5974108d20 msgr2=0x7f597419bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5974108d20 0x7f597419bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 -- 172.21.15.28:0/3605329236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 msgr2=0x7f597419b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f597419b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f596bfff640 1 -- 172.21.15.28:0/3605329236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59741a64b0 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.919+0000 7f597259c640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f597419b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f596bfff640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d770 tx=0x7f595c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c017070 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595c009c30 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59741a67a0 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c00f620 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.920+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59741a6d30 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.921+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5974103d10 con 0x7f59741083d0 2024-09-21T05:46:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.923+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f595c01a070 con 0x7f59741083d0 2024-09-21T05:46:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.923+0000 7f596b7fe640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 0x7f59600b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.924+0000 7f597259c640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 0x7f59600b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.924+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f595c0e6e50 con 0x7f59741083d0 2024-09-21T05:46:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.924+0000 7f597259c640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 0x7f59600b8af0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f597406c700 tx=0x7f596400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:15.927+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f595c009de0 con 0x7f59741083d0 2024-09-21T05:46:16.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.150+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5974102ef0 con 0x7f59741083d0 2024-09-21T05:46:16.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.151+0000 7f596b7fe640 1 -- 172.21.15.28:0/3605329236 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11986 (secure 0 0 0) 0x7f595c0b1a40 con 0x7f59741083d0 2024-09-21T05:46:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:16.153 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"btime":"2024-09-21T05:44:52:047664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:44:52.047653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:46:16.154 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-09-21T05:46:16.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.153+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 msgr2=0x7f59600b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:16.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.153+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 0x7f59600b8af0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f597406c700 tx=0x7f596400b390 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 msgr2=0x7f597419b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d770 tx=0x7f595c00dc30 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 shutdown_connections 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5974108d20 0x7f597419bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f59600b65e0 0x7f59600b8af0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59741083d0 0x7f597419b6e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 --2- 172.21.15.28:0/3605329236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5974102ae0 0x7f597419b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 >> 172.21.15.28:0/3605329236 conn(0x7f5974079760 msgr2=0x7f59741043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 shutdown_connections 2024-09-21T05:46:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:16.154+0000 7f5978a8e640 1 -- 172.21.15.28:0/3605329236 wait complete. 2024-09-21T05:46:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:16 smithi028 ceph-mon[31227]: pgmap v302: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T05:46:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:16 smithi028 ceph-mon[31227]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-09-21T05:46:16.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:16 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3605329236' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:16.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:46:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:16 smithi181 ceph-mon[34819]: pgmap v302: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T05:46:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:16 smithi181 ceph-mon[34819]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-09-21T05:46:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:16 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3605329236' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:16 smithi129 ceph-mon[34885]: pgmap v302: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T05:46:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:16 smithi129 ceph-mon[34885]: mgrmap e23: x(active, since 8m), standbys: z, y 2024-09-21T05:46:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:16 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3605329236' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 -- 172.21.15.28:0/3990092551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e81094a0 msgr2=0x7f49e81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/3990092551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e81094a0 0x7f49e81098b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009900 tx=0x7f49d802a410 comp rx=0 tx=0).stop 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 -- 172.21.15.28:0/3990092551 shutdown_connections 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/3990092551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8109df0 0x7f49e8110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/3990092551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e81094a0 0x7f49e81098b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/3990092551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e8103b60 0x7f49e8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.485+0000 7f49ef9b7640 1 -- 172.21.15.28:0/3990092551 >> 172.21.15.28:0/3990092551 conn(0x7f49e807b850 msgr2=0x7f49e807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.486+0000 7f49ef9b7640 1 -- 172.21.15.28:0/3990092551 shutdown_connections 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.486+0000 7f49ef9b7640 1 -- 172.21.15.28:0/3990092551 wait complete. 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.486+0000 7f49ef9b7640 1 Processor -- start 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.486+0000 7f49ef9b7640 1 -- start start 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8103b60 0x7f49e819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e81094a0 0x7f49e819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49e8113ab0 con 0x7f49e8109df0 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f49e8113930 con 0x7f49e8103b60 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f49e8113c30 con 0x7f49e81094a0 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44236/0 (socket says 172.21.15.28:44236) 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 -- 172.21.15.28:0/610048510 learned_addr learned my addr 172.21.15.28:0/610048510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ed753640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8103b60 0x7f49e819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ecf52640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e81094a0 0x7f49e819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 -- 172.21.15.28:0/610048510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e81094a0 msgr2=0x7f49e819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e81094a0 0x7f49e819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 -- 172.21.15.28:0/610048510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8103b60 msgr2=0x7f49e819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8103b60 0x7f49e819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 -- 172.21.15.28:0/610048510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e81a0370 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49edf54640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f49e000cce0 tx=0x7f49e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e000f450 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e000faa0 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.487+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e81a0660 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.488+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0018b70 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.488+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e81a6e80 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.489+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e8103f70 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.490+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f49e0018d10 con 0x7f49e8109df0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.491+0000 7f49de7fc640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 0x7f49d40b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.491+0000 7f49ed753640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 0x7f49d40b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.491+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f49e00e78c0 con 0x7f49e8109df0 2024-09-21T05:46:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.491+0000 7f49ed753640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 0x7f49d40b8b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49e4005d00 tx=0x7f49e400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.494+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f49e0014030 con 0x7f49e8109df0 2024-09-21T05:46:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.682+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49e810bc20 con 0x7f49e8109df0 2024-09-21T05:46:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.684+0000 7f49de7fc640 1 -- 172.21.15.28:0/610048510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+23122 (secure 0 0 0) 0x7f49e00ae100 con 0x7f49e8109df0 2024-09-21T05:46:17.686 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:17.686 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:14.248651+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:46:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 msgr2=0x7f49d40b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 0x7f49d40b8b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49e4005d00 tx=0x7f49e400b410 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 msgr2=0x7f49e819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f49e000cce0 tx=0x7f49e000a620 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 shutdown_connections 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f49d40b6610 0x7f49d40b8b20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e8109df0 0x7f49e819fd00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49e81094a0 0x7f49e819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 --2- 172.21.15.28:0/610048510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49e8103b60 0x7f49e819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 >> 172.21.15.28:0/610048510 conn(0x7f49e807b850 msgr2=0x7f49e8105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.686+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 shutdown_connections 2024-09-21T05:46:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:17.687+0000 7f49ef9b7640 1 -- 172.21.15.28:0/610048510 wait complete. 2024-09-21T05:46:18.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T05:44:39.761973+0000', 'modified': '2024-09-21T05:44:52.047653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24688, 'mds_1': 24680, 'mds_2': 24673, 'mds_3': 24664, 'mds_4': 24653}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14850': {'gid': 14850, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/2924711217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 2924711217}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 2924711217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14856': {'gid': 14856, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2765835727', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2765835727}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2765835727}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24614': {'gid': 24614, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6833/2193748712', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2193748712}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 2193748712}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24622': {'gid': 24622, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6833/1938433457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 1938433457}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 1938433457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24638': {'gid': 24638, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6835/2842004697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 2842004697}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2842004697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24653': {'gid': 24653, 'name': 'h', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.129:6837/2283221342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 2283221342}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 2283221342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24664': {'gid': 24664, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.181:6835/1316004847', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 1316004847}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 1316004847}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24673': {'gid': 24673, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.181:6837/2388771333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 2388771333}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2388771333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24680': {'gid': 24680, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6839/3631196486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 3631196486}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 3631196486}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24688': {'gid': 24688, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.181:6839/3430071220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3430071220}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3430071220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24688, 'qdb_cluster': [24653, 24664, 24673, 24680, 24688]} 2024-09-21T05:46:18.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T05:46:18.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T05:46:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:18 smithi028 ceph-mon[31227]: pgmap v303: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T05:46:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:18 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/610048510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:18 smithi181 ceph-mon[34819]: pgmap v303: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T05:46:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:18 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/610048510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:18 smithi129 ceph-mon[34885]: pgmap v303: 97 pgs: 97 active+clean; 647 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T05:46:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:18 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/610048510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 -- 172.21.15.28:0/3326647352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44102a60 msgr2=0x7f7b44102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 --2- 172.21.15.28:0/3326647352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44102a60 0x7f7b44102e70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009ca0 tx=0x7f7b4002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 -- 172.21.15.28:0/3326647352 shutdown_connections 2024-09-21T05:46:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 --2- 172.21.15.28:0/3326647352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b44108cf0 0x7f7b4410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 --2- 172.21.15.28:0/3326647352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b441083a0 0x7f7b441087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 --2- 172.21.15.28:0/3326647352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44102a60 0x7f7b44102e70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 -- 172.21.15.28:0/3326647352 >> 172.21.15.28:0/3326647352 conn(0x7f7b44079730 msgr2=0x7f7b44079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 -- 172.21.15.28:0/3326647352 shutdown_connections 2024-09-21T05:46:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.984+0000 7f7b4c009640 1 -- 172.21.15.28:0/3326647352 wait complete. 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 Processor -- start 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 -- start start 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 0x7f7b4419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b441083a0 0x7f7b4419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44113a80 con 0x7f7b44108cf0 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44113900 con 0x7f7b441083a0 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4c009640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44113c00 con 0x7f7b44102a60 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4a5a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4a5a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44264/0 (socket says 172.21.15.28:44264) 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b4a5a6640 1 -- 172.21.15.28:0/461264086 learned_addr learned my addr 172.21.15.28:0/461264086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.985+0000 7f7b495a4640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b441083a0 0x7f7b4419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b49da5640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 0x7f7b4419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 -- 172.21.15.28:0/461264086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 msgr2=0x7f7b4419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 0x7f7b4419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 -- 172.21.15.28:0/461264086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b441083a0 msgr2=0x7f7b4419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b441083a0 0x7f7b4419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 -- 172.21.15.28:0/461264086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b441a2560 con 0x7f7b44108cf0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b49da5640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 0x7f7b4419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4a5a6640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00d750 tx=0x7f7b3c00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b3affd640 1 -- 172.21.15.28:0/461264086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c017070 con 0x7f7b44108cf0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b3affd640 1 -- 172.21.15.28:0/461264086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c0097c0 con 0x7f7b44108cf0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b441a2850 con 0x7f7b44108cf0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b3affd640 1 -- 172.21.15.28:0/461264086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c004ba0 con 0x7f7b44108cf0 2024-09-21T05:46:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.986+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b441a9070 con 0x7f7b44108cf0 2024-09-21T05:46:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.989+0000 7f7b5257d740 1 Processor -- start 2024-09-21T05:46:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b3affd640 1 -- 172.21.15.28:0/461264086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f7b3c009c90 con 0x7f7b44108cf0 2024-09-21T05:46:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 -- start start 2024-09-21T05:46:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 0x5572ee66e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee66e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572ee5e5c10 con 0x5572ee2c7f60 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572ee5d5040 con 0x5572ee29f0a0 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b5257d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572ee5d8e50 con 0x5572ee2a7280 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b495a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b495a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44278/0 (socket says 172.21.15.28:44278) 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.990+0000 7f7b495a4640 1 -- 172.21.15.28:0/1573721201 learned_addr learned my addr 172.21.15.28:0/1573721201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee66e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 0x5572ee66e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 -- 172.21.15.28:0/1573721201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 msgr2=0x5572ee66e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b3affd640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 0x7f7b300b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee66e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 -- 172.21.15.28:0/1573721201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 msgr2=0x5572ee66e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 0x5572ee66e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 -- 172.21.15.28:0/1573721201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572ee333540 con 0x5572ee2c7f60 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 0x5572ee66e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b4a5a6640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 0x7f7b300b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400afd0 tx=0x7f7b3400a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f462779f87b0ff6b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b3affd640 1 -- 172.21.15.28:0/461264086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7b3c0e6dc0 con 0x7f7b44108cf0 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b2ffff640 1 -- 172.21.15.28:0/1573721201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34019030 con 0x5572ee2c7f60 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b2ffff640 1 -- 172.21.15.28:0/1573721201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b340110d0 con 0x5572ee2c7f60 2024-09-21T05:46:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.991+0000 7f7b2ffff640 1 -- 172.21.15.28:0/1573721201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400f780 con 0x5572ee2c7f60 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 -- 172.21.15.28:0/1573721201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 msgr2=0x5572ee66df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b4a5a6640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 0x7f7b300b8b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009610 tx=0x7f7b3c0d0040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400afd0 tx=0x7f7b3400a7c0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 -- 172.21.15.28:0/1573721201 shutdown_connections 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee66e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee29f0a0 0x5572ee66e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1573721201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2c7f60 0x5572ee66df00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 -- 172.21.15.28:0/1573721201 >> 172.21.15.28:0/1573721201 conn(0x5572ee2bc350 msgr2=0x5572ee66bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 -- 172.21.15.28:0/1573721201 shutdown_connections 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.992+0000 7f7b5257d740 1 -- 172.21.15.28:0/1573721201 wait complete. 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 -- start start 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 Processor -- start 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee67a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572ee4365f0 con 0x5572ee29f0a0 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572ee570010 con 0x5572ee2c7f60 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572ee67c6e0 con 0x5572ee2a7280 2024-09-21T05:46:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.993+0000 7f7b5257d740 10 client.0 _ll_register_callbacks cb 0x5572ee5cf340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44284/0 (socket says 172.21.15.28:44284) 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 -- 172.21.15.28:0/4261381559 learned_addr learned my addr 172.21.15.28:0/4261381559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b49da5640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee67a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 -- 172.21.15.28:0/4261381559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 msgr2=0x5572ee67a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee67a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 -- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 msgr2=0x5572ee6859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 -- 172.21.15.28:0/4261381559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572ee59da00 con 0x5572ee29f0a0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b495a4640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400a670 tx=0x7f7b34005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a67f655b51c4822 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34019040 con 0x5572ee29f0a0 2024-09-21T05:46:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7b34011930 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b340110d0 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7b34011b30 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.994+0000 7f7b5257d740 10 client.15369 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5572ee6886f0 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b340110d0 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f7b3402e020 con 0x5572ee29f0a0 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 10 client.15369 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 10 client.15369 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T05:46:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 20 client.15369 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee2868e0 0x5572ee68bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 4 client.15369 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572ee5e7b60 con 0x5572ee2868e0 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.995+0000 7f7b49da5640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee2868e0 0x5572ee68bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.996+0000 7f7b2d7fa640 10 client.15369 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:46:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.996+0000 7f7b49da5640 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee2868e0 0x5572ee68bcc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be38024c25609fec server_cookie=c9344607e8f73f2 in_seq=0 out_seq=0 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/4261381559 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5572ee5e7b60 con 0x5572ee2868e0 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b2d7fa640 10 client.15369 handle_command_reply: tid=0 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 client.15369 shutdown 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 msgr2=0x5572ee679b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400a670 tx=0x7f7b34005740 comp rx=0 tx=0).stop 2024-09-21T05:46:19.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 shutdown_connections 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee2868e0 0x5572ee68bcc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee2a7280 0x5572ee67a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b2d7fa640 0 client.15369 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 --2- 172.21.15.28:0/4261381559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee29f0a0 0x5572ee679b00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 >> 172.21.15.28:0/4261381559 conn(0x5572ee2bc350 msgr2=0x5572ee6710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 shutdown_connections 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 1 -- 172.21.15.28:0/4261381559 wait complete. 2024-09-21T05:46:19.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:18.997+0000 7f7b5257d740 20 client.15369 trim_cache size 0 max 16384 2024-09-21T05:46:19.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.015+0000 7f7b5257d740 1 Processor -- start 2024-09-21T05:46:19.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.015+0000 7f7b5257d740 1 -- start start 2024-09-21T05:46:19.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee685940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6736c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68a000 con 0x5572ee2a7280 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68a6d0 con 0x5572ee2c7f60 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b5257d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68ada0 con 0x5572ee29f0a0 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 -- 172.21.15.28:0/1706138914 learned_addr learned my addr 172.21.15.28:0/1706138914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee685940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 -- 172.21.15.28:0/1706138914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 msgr2=0x5572ee685940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee685940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 -- 172.21.15.28:0/1706138914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 msgr2=0x5572ee6736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6736c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 -- 172.21.15.28:0/1706138914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572ee666080 con 0x5572ee2a7280 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7b400093d0 tx=0x7f7b40031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9343cc21c310ca6e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.016+0000 7f7b2f7fe640 1 -- 172.21.15.28:0/1706138914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40031de0 con 0x5572ee2a7280 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b2f7fe640 1 -- 172.21.15.28:0/1706138914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40036040 con 0x5572ee2a7280 2024-09-21T05:46:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b2f7fe640 1 -- 172.21.15.28:0/1706138914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b400042a0 con 0x5572ee2a7280 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 -- 172.21.15.28:0/1706138914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 msgr2=0x5572ee673180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7b400093d0 tx=0x7f7b40031c30 comp rx=0 tx=0).stop 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 -- 172.21.15.28:0/1706138914 shutdown_connections 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee6736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee673180 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1706138914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee685940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 -- 172.21.15.28:0/1706138914 >> 172.21.15.28:0/1706138914 conn(0x5572ee2bc350 msgr2=0x5572ee666ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.017+0000 7f7b5257d740 1 -- 172.21.15.28:0/1706138914 shutdown_connections 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 -- 172.21.15.28:0/1706138914 wait complete. 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 -- start start 2024-09-21T05:46:19.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 Processor -- start 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee694660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee66b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68bb40 con 0x5572ee2a7280 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68b470 con 0x5572ee2c7f60 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572ee68c210 con 0x5572ee29f0a0 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.018+0000 7f7b5257d740 10 client.0 _ll_register_callbacks cb 0x7f7b34013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44302/0 (socket says 172.21.15.28:44302) 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 -- 172.21.15.28:0/1409984383 learned_addr learned my addr 172.21.15.28:0/1409984383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee694660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee66b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 -- 172.21.15.28:0/1409984383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 msgr2=0x5572ee694660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee694660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 -- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 msgr2=0x5572ee66b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee66b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 -- 172.21.15.28:0/1409984383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572ee67a4a0 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee694660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b49da5640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7b40003fd0 tx=0x7f7b400095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bef23d4d277af0f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40031450 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7b400315f0 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40036040 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7b400317f0 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b5257d740 10 client.15375 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.019+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5572ee68f070 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40036040 con 0x5572ee2a7280 2024-09-21T05:46:19.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f7b4002f660 con 0x5572ee2a7280 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 10 client.15375 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 10 client.15375 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 20 client.15375 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee25bc90 0x5572ee6fa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 4 client.15375 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "status"}) -- 0x5572ee5e3610 con 0x5572ee25bc90 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.020+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee25bc90 0x5572ee6fa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b495a4640 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee25bc90 0x5572ee6fa350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c67d80939915bfa server_cookie=e59da10188f80c99 in_seq=0 out_seq=0 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b2d7fa640 10 client.15375 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b2d7fa640 1 -- 172.21.15.28:0/1409984383 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5572ee5e3610 con 0x5572ee25bc90 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b2d7fa640 10 client.15375 handle_command_reply: tid=0 2024-09-21T05:46:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 client.15375 shutdown 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 msgr2=0x5572ee66b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7b40003fd0 tx=0x7f7b400095d0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 shutdown_connections 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5572ee25bc90 0x5572ee6fa350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572ee2c7f60 0x5572ee66b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572ee2a7280 0x5572ee66b380 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b2d7fa640 0 client.15375 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 --2- 172.21.15.28:0/1409984383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572ee29f0a0 0x5572ee694660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 >> 172.21.15.28:0/1409984383 conn(0x5572ee2bc350 msgr2=0x5572ee674a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 shutdown_connections 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 1 -- 172.21.15.28:0/1409984383 wait complete. 2024-09-21T05:46:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.021+0000 7f7b5257d740 20 client.15375 trim_cache size 0 max 16384 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24653, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 87.820569735999996, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 120.399338317 2024-09-21T05:46:19.029 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 msgr2=0x7f7b300b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 0x7f7b300b8b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009610 tx=0x7f7b3c0d0040 comp rx=0 tx=0).stop 2024-09-21T05:46:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 msgr2=0x7f7b441a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00d750 tx=0x7f7b3c00db50 comp rx=0 tx=0).stop 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 shutdown_connections 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7b300b6610 0x7f7b300b8b20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b44108cf0 0x7f7b441a1ef0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b441083a0 0x7f7b4419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 --2- 172.21.15.28:0/461264086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b44102a60 0x7f7b4419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 >> 172.21.15.28:0/461264086 conn(0x7f7b44079730 msgr2=0x7f7b4410e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 shutdown_connections 2024-09-21T05:46:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:19.022+0000 7f7b4c009640 1 -- 172.21.15.28:0/461264086 wait complete. 2024-09-21T05:46:19.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.f status 2024-09-21T05:46:20.030 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 -- 172.21.15.28:0/3327398573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 msgr2=0x7f9124110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 --2- 172.21.15.28:0/3327398573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124110640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f911c009ca0 tx=0x7f911c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 -- 172.21.15.28:0/3327398573 shutdown_connections 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 --2- 172.21.15.28:0/3327398573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124110640 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 --2- 172.21.15.28:0/3327398573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91241094c0 0x7f91241098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 --2- 172.21.15.28:0/3327398573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9124103b80 0x7f9124103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 -- 172.21.15.28:0/3327398573 >> 172.21.15.28:0/3327398573 conn(0x7f91240ff5d0 msgr2=0x7f9124101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 -- 172.21.15.28:0/3327398573 shutdown_connections 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.510+0000 7f912b279640 1 -- 172.21.15.28:0/3327398573 wait complete. 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.511+0000 7f912b279640 1 Processor -- start 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.511+0000 7f912b279640 1 -- start start 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.511+0000 7f912b279640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9124103b80 0x7f9124075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.511+0000 7f912b279640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91241094c0 0x7f9124075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f912b279640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f912b279640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9124113b30 con 0x7f9124109e10 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f912b279640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91241139b0 con 0x7f9124103b80 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f912b279640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9124113cb0 con 0x7f91241094c0 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44320/0 (socket says 172.21.15.28:44320) 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 -- 172.21.15.28:0/3341791269 learned_addr learned my addr 172.21.15.28:0/3341791269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129015640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9124103b80 0x7f9124075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9128814640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91241094c0 0x7f9124075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 -- 172.21.15.28:0/3341791269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91241094c0 msgr2=0x7f9124075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91241094c0 0x7f9124075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 -- 172.21.15.28:0/3341791269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9124103b80 msgr2=0x7f9124075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9124103b80 0x7f9124075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 -- 172.21.15.28:0/3341791269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91241a8760 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f9129816640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f911c005cc0 tx=0x7f911c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f911a7fc640 1 -- 172.21.15.28:0/3341791269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c003bf0 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f911a7fc640 1 -- 172.21.15.28:0/3341791269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f911c003d90 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f911a7fc640 1 -- 172.21.15.28:0/3341791269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c02fb20 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.512+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91241a89f0 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.513+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91241a9000 con 0x7f9124109e10 2024-09-21T05:46:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.515+0000 7f91317ed740 1 Processor -- start 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f911a7fc640 1 -- 172.21.15.28:0/3341791269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f911c004580 con 0x7f9124109e10 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 -- start start 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d458f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d459980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d45d3d0c10 con 0x55d45d08a0a0 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d45d3c0040 con 0x55d45d0b2f60 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f91317ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d45d3c3e50 con 0x55d45d092280 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44322/0 (socket says 172.21.15.28:44322) 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.516+0000 7f9129015640 1 -- 172.21.15.28:0/1623502149 learned_addr learned my addr 172.21.15.28:0/1623502149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9128814640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d458f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9128814640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d459980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 -- 172.21.15.28:0/1623502149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 msgr2=0x55d45d459980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d459980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 -- 172.21.15.28:0/1623502149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 msgr2=0x55d45d458f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d458f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 -- 172.21.15.28:0/1623502149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d45d11e540 con 0x55d45d08a0a0 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9128814640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d459980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f911a7fc640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 0x7f91100b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9129015640 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9120009a40 tx=0x7f9120009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353d6612cefbf467 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f9128814640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 0x7f91100b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f911a7fc640 1 -- 172.21.15.28:0/3341791269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f911c1086c0 con 0x7f9124109e10 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f910f7fe640 1 -- 172.21.15.28:0/1623502149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120017070 con 0x55d45d08a0a0 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f910f7fe640 1 -- 172.21.15.28:0/1623502149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9120013470 con 0x55d45d08a0a0 2024-09-21T05:46:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.517+0000 7f910f7fe640 1 -- 172.21.15.28:0/1623502149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120012630 con 0x55d45d08a0a0 2024-09-21T05:46:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 -- 172.21.15.28:0/1623502149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 msgr2=0x55d45d459440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9120009a40 tx=0x7f9120009f00 comp rx=0 tx=0).stop 2024-09-21T05:46:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f9128814640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 0x7f91100b8af0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9124108e10 tx=0x7f9114014040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 -- 172.21.15.28:0/1623502149 shutdown_connections 2024-09-21T05:46:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d459980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d459440 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 --2- 172.21.15.28:0/1623502149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d458f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.518+0000 7f91317ed740 1 -- 172.21.15.28:0/1623502149 >> 172.21.15.28:0/1623502149 conn(0x55d45d0a7350 msgr2=0x55d45d456df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 -- 172.21.15.28:0/1623502149 shutdown_connections 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 -- 172.21.15.28:0/1623502149 wait complete. 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 -- start start 2024-09-21T05:46:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 Processor -- start 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d464cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.519+0000 7f91317ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d470a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f91317ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d45d2215f0 con 0x55d45d08a0a0 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f91317ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d45d35b010 con 0x55d45d0b2f60 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f91317ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d45d4676e0 con 0x55d45d092280 2024-09-21T05:46:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f91317ed740 10 client.0 _ll_register_callbacks cb 0x55d45d3ba340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44332/0 (socket says 172.21.15.28:44332) 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 -- 172.21.15.28:0/4012064675 learned_addr learned my addr 172.21.15.28:0/4012064675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129816640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d470a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9128814640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d464cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 -- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 msgr2=0x55d45d464cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d464cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 -- 172.21.15.28:0/4012064675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 msgr2=0x55d45d470a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d470a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 -- 172.21.15.28:0/4012064675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d45d388a00 con 0x55d45d08a0a0 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f9129015640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9120013d80 tx=0x7f9120013620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81050fd0e29093a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f910cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.520+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120017050 con 0x55d45d08a0a0 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f912001c450 con 0x55d45d08a0a0 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9120013470 con 0x55d45d08a0a0 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f912001c650 con 0x55d45d08a0a0 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 10 client.15387 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d45d473510 con 0x55d45d08a0a0 2024-09-21T05:46:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9120013470 con 0x55d45d08a0a0 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f912002d020 con 0x55d45d08a0a0 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 10 client.15387 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 10 client.15387 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 20 client.15387 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d0718e0 0x55d45d476c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 4 client.15387 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.521+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d45d3d2b60 con 0x55d45d0718e0 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.522+0000 7f9128814640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d0718e0 0x55d45d476c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.522+0000 7f9128814640 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d0718e0 0x55d45d476c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86bb93feeef69a75 server_cookie=ef8b8686a42f7e3a in_seq=0 out_seq=0 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.522+0000 7f910cff9640 10 client.15387 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f910cff9640 1 -- 172.21.15.28:0/4012064675 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d45d3d2b60 con 0x55d45d0718e0 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f910cff9640 10 client.15387 handle_command_reply: tid=0 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 client.15387 shutdown 2024-09-21T05:46:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 msgr2=0x55d45d464750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9120013d80 tx=0x7f9120013620 comp rx=0 tx=0).stop 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 shutdown_connections 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d0718e0 0x55d45d476c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d470a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d464cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 --2- 172.21.15.28:0/4012064675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d464750 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 >> 172.21.15.28:0/4012064675 conn(0x55d45d0a7350 msgr2=0x55d45d45c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f910cff9640 0 client.15387 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:46:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 shutdown_connections 2024-09-21T05:46:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.523+0000 7f91317ed740 1 -- 172.21.15.28:0/4012064675 wait complete. 2024-09-21T05:46:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.524+0000 7f91317ed740 20 client.15387 trim_cache size 0 max 16384 2024-09-21T05:46:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 Processor -- start 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 -- start start 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d08a0a0 0x55d45d45f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 0x55d45d4709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d45d474f80 con 0x55d45d0b2f60 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d45d475650 con 0x55d45d092280 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f91317ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d45d475d20 con 0x55d45d08a0a0 2024-09-21T05:46:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44342/0 (socket says 172.21.15.28:44342) 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 -- 172.21.15.28:0/2381618914 learned_addr learned my addr 172.21.15.28:0/2381618914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129015640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d08a0a0 0x55d45d45f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9128814640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 0x55d45d4709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 -- 172.21.15.28:0/2381618914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d08a0a0 msgr2=0x55d45d45f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d08a0a0 0x55d45d45f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 -- 172.21.15.28:0/2381618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 msgr2=0x55d45d4709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 0x55d45d4709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9129816640 1 -- 172.21.15.28:0/2381618914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d45d457ad0 con 0x55d45d0b2f60 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.542+0000 7f9128814640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 0x55d45d4709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.543+0000 7f9129816640 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f911c0cb9c0 tx=0x7f911c0caed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e65572dc4e82e8a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.543+0000 7f910effd640 1 -- 172.21.15.28:0/2381618914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c0f4020 con 0x55d45d0b2f60 2024-09-21T05:46:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.543+0000 7f910effd640 1 -- 172.21.15.28:0/2381618914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f911c036040 con 0x55d45d0b2f60 2024-09-21T05:46:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.543+0000 7f910effd640 1 -- 172.21.15.28:0/2381618914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c0d8170 con 0x55d45d0b2f60 2024-09-21T05:46:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- 172.21.15.28:0/2381618914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 msgr2=0x55d45d470ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f911c0cb9c0 tx=0x7f911c0caed0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- 172.21.15.28:0/2381618914 shutdown_connections 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d0b2f60 0x55d45d470ee0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d092280 0x55d45d4709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 --2- 172.21.15.28:0/2381618914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d08a0a0 0x55d45d45f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- 172.21.15.28:0/2381618914 >> 172.21.15.28:0/2381618914 conn(0x55d45d0a7350 msgr2=0x55d45d45f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- 172.21.15.28:0/2381618914 shutdown_connections 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- 172.21.15.28:0/2381618914 wait complete. 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 -- start start 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.544+0000 7f91317ed740 1 Processor -- start 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d478bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d47dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d45d476ac0 con 0x55d45d08a0a0 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d45d4763f0 con 0x55d45d0b2f60 2024-09-21T05:46:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d45d477190 con 0x55d45d092280 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f91317ed740 10 client.0 _ll_register_callbacks cb 0x7f9120009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44358/0 (socket says 172.21.15.28:44358) 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 -- 172.21.15.28:0/698196764 learned_addr learned my addr 172.21.15.28:0/698196764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9128814640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d478bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129816640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d47dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 -- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 msgr2=0x55d45d478bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d478bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 -- 172.21.15.28:0/698196764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 msgr2=0x55d45d47dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d47dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 -- 172.21.15.28:0/698196764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d45d47e2a0 con 0x55d45d08a0a0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129816640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d47dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f9129015640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9120013d80 tx=0x7f91200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d1a4dcf7f9004 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.545+0000 7f910cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120017070 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f912002f670 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f911c036040 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9120013470 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 10 client.15393 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d45d47f5d0 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f911c036040 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f912002d040 con 0x55d45d08a0a0 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 10 client.15393 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 10 client.15393 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 20 client.15393 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d046c90 0x55d45d4e51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 4 client.15393 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.546+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "status"}) -- 0x55d45d3ce610 con 0x55d45d046c90 2024-09-21T05:46:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f9128814640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d046c90 0x55d45d4e51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f9128814640 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d046c90 0x55d45d4e51b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1a250268feeec9d2 server_cookie=92739164f8d117f6 in_seq=0 out_seq=0 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f910cff9640 10 client.15393 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f910cff9640 1 -- 172.21.15.28:0/698196764 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d45d3ce610 con 0x55d45d046c90 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f910cff9640 10 client.15393 handle_command_reply: tid=0 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.547+0000 7f91317ed740 1 client.15393 shutdown 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 msgr2=0x55d45d478630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9120013d80 tx=0x7f91200046c0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 shutdown_connections 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55d45d046c90 0x55d45d4e51b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d45d0b2f60 0x55d45d47dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d45d092280 0x55d45d478bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 --2- 172.21.15.28:0/698196764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d45d08a0a0 0x55d45d478630 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f910cff9640 0 client.15393 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 >> 172.21.15.28:0/698196764 conn(0x55d45d0a7350 msgr2=0x55d45d464500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 shutdown_connections 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 1 -- 172.21.15.28:0/698196764 wait complete. 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f91317ed740 20 client.15393 trim_cache size 0 max 16384 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24664, 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 90.360444228999995, 2024-09-21T05:46:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 126.740849087 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 msgr2=0x7f91100b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 0x7f91100b8af0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9124108e10 tx=0x7f9114014040 comp rx=0 tx=0).stop 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 msgr2=0x7f9124076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.548+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f911c005cc0 tx=0x7f911c02f380 comp rx=0 tx=0).stop 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 shutdown_connections 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91100b65e0 0x7f91100b8af0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9124109e10 0x7f9124076190 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91241094c0 0x7f9124075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 --2- 172.21.15.28:0/3341791269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9124103b80 0x7f9124075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 >> 172.21.15.28:0/3341791269 conn(0x7f91240ff5d0 msgr2=0x7f91241019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 shutdown_connections 2024-09-21T05:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:20.549+0000 7f912b279640 1 -- 172.21.15.28:0/3341791269 wait complete. 2024-09-21T05:46:20.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:20 smithi181 ceph-mon[34819]: pgmap v304: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:46:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:20 smithi028 ceph-mon[31227]: pgmap v304: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:46:20.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:20 smithi129 ceph-mon[34885]: pgmap v304: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:46:21.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T05:46:21.626 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 -- 172.21.15.28:0/1723239747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 msgr2=0x7fe8a0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1723239747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a0110330 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe898009ca0 tx=0x7fe89802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 -- 172.21.15.28:0/1723239747 shutdown_connections 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1723239747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a0110330 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1723239747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a0105330 0x7fe8a010d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1723239747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a01049e0 0x7fe8a0104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 -- 172.21.15.28:0/1723239747 >> 172.21.15.28:0/1723239747 conn(0x7fe8a0079750 msgr2=0x7fe8a0079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 -- 172.21.15.28:0/1723239747 shutdown_connections 2024-09-21T05:46:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.087+0000 7fe8a5399640 1 -- 172.21.15.28:0/1723239747 wait complete. 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.088+0000 7fe8a5399640 1 Processor -- start 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.088+0000 7fe8a5399640 1 -- start start 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a01049e0 0x7fe8a019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a0105330 0x7fe8a019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0113aa0 con 0x7fe8a010de40 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0113920 con 0x7fe8a0105330 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe8a5399640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0113c20 con 0x7fe8a01049e0 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89effd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a01049e0 0x7fe8a019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44376/0 (socket says 172.21.15.28:44376) 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a0105330 0x7fe8a019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89f7fe640 1 -- 172.21.15.28:0/1589163457 learned_addr learned my addr 172.21.15.28:0/1589163457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89f7fe640 1 -- 172.21.15.28:0/1589163457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a01049e0 msgr2=0x7fe8a019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.089+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a01049e0 0x7fe8a019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe89f7fe640 1 -- 172.21.15.28:0/1589163457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a0105330 msgr2=0x7fe8a019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a0105330 0x7fe8a019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe89f7fe640 1 -- 172.21.15.28:0/1589163457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a01a03e0 con 0x7fe8a010de40 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe898005cc0 tx=0x7fe89802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe897fff640 1 -- 172.21.15.28:0/1589163457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898003bf0 con 0x7fe8a010de40 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a01a0670 con 0x7fe8a010de40 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe897fff640 1 -- 172.21.15.28:0/1589163457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe898003d90 con 0x7fe8a010de40 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe897fff640 1 -- 172.21.15.28:0/1589163457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89802fbc0 con 0x7fe8a010de40 2024-09-21T05:46:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.090+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a01a6e30 con 0x7fe8a010de40 2024-09-21T05:46:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.093+0000 7fe8ab90d740 1 Processor -- start 2024-09-21T05:46:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.093+0000 7fe897fff640 1 -- 172.21.15.28:0/1589163457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7fe89802fd60 con 0x7fe8a010de40 2024-09-21T05:46:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.093+0000 7fe8ab90d740 1 -- start start 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x56130210a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x56130210a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561302081c10 con 0x561301d63f60 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561302071040 con 0x561301d3b0a0 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561302074e50 con 0x561301d43280 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44380/0 (socket says 172.21.15.28:44380) 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2498754413 learned_addr learned my addr 172.21.15.28:0/2498754413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x56130210a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89effd640 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x56130210a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2498754413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 msgr2=0x56130210a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x56130210a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2498754413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 msgr2=0x56130210a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x56130210a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.094+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2498754413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561301dcf540 con 0x561301d63f60 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe8900124c0 tx=0x7fe89000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6d7ad2497fa585 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe897fff640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 0x7fe88c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe8957fa640 1 -- 172.21.15.28:0/2498754413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890004030 con 0x561301d63f60 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe8957fa640 1 -- 172.21.15.28:0/2498754413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8900054d0 con 0x561301d63f60 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe89effd640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 0x7fe88c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe8957fa640 1 -- 172.21.15.28:0/2498754413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890004b60 con 0x561301d63f60 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.095+0000 7fe897fff640 1 -- 172.21.15.28:0/1589163457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe8981088c0 con 0x7fe8a010de40 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2498754413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 msgr2=0x561302109f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe89effd640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 0x7fe88c0b8a70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x561302108b80 tx=0x7fe88800a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe8900124c0 tx=0x7fe89000b820 comp rx=0 tx=0).stop 2024-09-21T05:46:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2498754413 shutdown_connections 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x56130210a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x56130210a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2498754413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302109f00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2498754413 >> 172.21.15.28:0/2498754413 conn(0x561301d58350 msgr2=0x561302107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2498754413 shutdown_connections 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.096+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2498754413 wait complete. 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 -- start start 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 Processor -- start 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x5613021159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x561302115f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561301ed25f0 con 0x561301d63f60 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56130200c010 con 0x561301d3b0a0 2024-09-21T05:46:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5613021186e0 con 0x561301d43280 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe8ab90d740 10 client.0 _ll_register_callbacks cb 0x56130206b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44394/0 (socket says 172.21.15.28:44394) 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe89f7fe640 1 -- 172.21.15.28:0/2264113822 learned_addr learned my addr 172.21.15.28:0/2264113822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe89effd640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x5613021159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.097+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x561302115f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 -- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 msgr2=0x561302115f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x561302115f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 -- 172.21.15.28:0/2264113822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 msgr2=0x5613021159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x5613021159c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 -- 172.21.15.28:0/2264113822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561302039a00 con 0x561301d63f60 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe8980caf20 tx=0x7fe8980ce2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74743d8f25470ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8867fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8980ce460 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe8980cdbb0 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8900054d0 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe8980cf050 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8ab90d740 10 client.15405 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561302124630 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.098+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8900054d0 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe8980ca2a0 con 0x561301d63f60 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 10 client.15405 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 10 client.15405 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 20 client.15405 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301d228e0 0x5613021280d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 4 client.15405 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561302083b60 con 0x561301d228e0 2024-09-21T05:46:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.099+0000 7fe89effd640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301d228e0 0x5613021280d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.100+0000 7fe8867fc640 10 client.15405 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.100+0000 7fe89effd640 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301d228e0 0x5613021280d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d4bdd6a0a3d0768 server_cookie=f2367bb765127421 in_seq=0 out_seq=0 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.100+0000 7fe8867fc640 1 -- 172.21.15.28:0/2264113822 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561302083b60 con 0x561301d228e0 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.100+0000 7fe8867fc640 10 client.15405 handle_command_reply: tid=0 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.100+0000 7fe8ab90d740 1 client.15405 shutdown 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 msgr2=0x561302121a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe8980caf20 tx=0x7fe8980ce2a0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 shutdown_connections 2024-09-21T05:46:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301d228e0 0x5613021280d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121a30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x561302115f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2264113822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x5613021159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 >> 172.21.15.28:0/2264113822 conn(0x561301d58350 msgr2=0x56130210d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8867fc640 0 client.15405 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 shutdown_connections 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2264113822 wait complete. 2024-09-21T05:46:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.101+0000 7fe8ab90d740 20 client.15405 trim_cache size 0 max 16384 2024-09-21T05:46:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 Processor -- start 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 -- start start 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x5613021219a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561302126410 con 0x561301d63f60 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561302126ae0 con 0x561301d3b0a0 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5613021271b0 con 0x561301d43280 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe89effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe89effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39006/0 (socket says 172.21.15.28:39006) 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.137+0000 7fe89effd640 1 -- 172.21.15.28:0/2293756007 learned_addr learned my addr 172.21.15.28:0/2293756007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x5613021219a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 -- 172.21.15.28:0/2293756007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 msgr2=0x5613021219a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x5613021219a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 -- 172.21.15.28:0/2293756007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 msgr2=0x561302121ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 -- 172.21.15.28:0/2293756007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613021185c0 con 0x561301d3b0a0 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe89effd640 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe88800a9c0 tx=0x7fe888015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67cdafe8427a43c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe894ff9640 1 -- 172.21.15.28:0/2293756007 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888004220 con 0x561301d3b0a0 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe894ff9640 1 -- 172.21.15.28:0/2293756007 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8880043c0 con 0x561301d3b0a0 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.138+0000 7fe894ff9640 1 -- 172.21.15.28:0/2293756007 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888028970 con 0x561301d3b0a0 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2293756007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 msgr2=0x561302110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe88800a9c0 tx=0x7fe888015000 comp rx=0 tx=0).stop 2024-09-21T05:46:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2293756007 shutdown_connections 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d63f60 0x561302121ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d43280 0x5613021219a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2293756007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d3b0a0 0x561302110020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2293756007 >> 172.21.15.28:0/2293756007 conn(0x561301d58350 msgr2=0x5613021109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2293756007 shutdown_connections 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.139+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2293756007 wait complete. 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 -- start start 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 Processor -- start 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d3b0a0 0x561302130b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d63f60 0x56130212a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561302127f50 con 0x561301d43280 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561302127880 con 0x561301d63f60 2024-09-21T05:46:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561302128620 con 0x561301d3b0a0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe8ab90d740 10 client.0 _ll_register_callbacks cb 0x7fe898009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44412/0 (socket says 172.21.15.28:44412) 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.140+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2946236020 learned_addr learned my addr 172.21.15.28:0/2946236020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89f7fe640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d63f60 0x56130212a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89effd640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d3b0a0 0x561302130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d3b0a0 msgr2=0x561302130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d3b0a0 0x561302130b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2946236020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d63f60 msgr2=0x56130212a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d63f60 0x56130212a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 -- 172.21.15.28:0/2946236020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130212ace0 con 0x561301d43280 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe89e7fc640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe890013fd0 tx=0x7fe890012300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b34705cf2a8d1e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8867fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890005460 con 0x561301d43280 2024-09-21T05:46:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8880043c0 con 0x561301d43280 2024-09-21T05:46:22.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe890004760 con 0x561301d43280 2024-09-21T05:46:22.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe890005600 con 0x561301d43280 2024-09-21T05:46:22.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8ab90d740 10 client.15411 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:22.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.141+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5613021935a0 con 0x561301d43280 2024-09-21T05:46:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8880043c0 con 0x561301d43280 2024-09-21T05:46:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe890028070 con 0x561301d43280 2024-09-21T05:46:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 10 client.15411 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 10 client.15411 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 20 client.15411 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301cf7c90 0x561302196b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 4 client.15411 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "status"}) -- 0x56130207f610 con 0x561301cf7c90 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.142+0000 7fe89effd640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301cf7c90 0x561302196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8867fc640 10 client.15411 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe89effd640 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301cf7c90 0x561302196b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6e9c63f1805b3203 server_cookie=1656f6d1f5712745 in_seq=0 out_seq=0 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8867fc640 1 -- 172.21.15.28:0/2946236020 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56130207f610 con 0x561301cf7c90 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8867fc640 10 client.15411 handle_command_reply: tid=0 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 client.15411 shutdown 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 msgr2=0x56130212a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe890013fd0 tx=0x7fe890012300 comp rx=0 tx=0).stop 2024-09-21T05:46:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 shutdown_connections 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561301cf7c90 0x561302196b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561301d63f60 0x56130212a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561301d43280 0x56130212a030 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 --2- 172.21.15.28:0/2946236020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561301d3b0a0 0x561302130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 >> 172.21.15.28:0/2946236020 conn(0x561301d58350 msgr2=0x561302107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8867fc640 0 client.15411 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 shutdown_connections 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.143+0000 7fe8ab90d740 1 -- 172.21.15.28:0/2946236020 wait complete. 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8ab90d740 20 client.15411 trim_cache size 0 max 16384 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:22.149 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24673, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 92.964082110000007, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 125.957562451 2024-09-21T05:46:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 msgr2=0x7fe88c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 0x7fe88c0b8a70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x561302108b80 tx=0x7fe88800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 msgr2=0x7fe8a019fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe898005cc0 tx=0x7fe89802f380 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 shutdown_connections 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe88c0b6560 0x7fe88c0b8a70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a010de40 0x7fe8a019fcb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a0105330 0x7fe8a019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 --2- 172.21.15.28:0/1589163457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a01049e0 0x7fe8a019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 >> 172.21.15.28:0/1589163457 conn(0x7fe8a0079750 msgr2=0x7fe8a0107960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 shutdown_connections 2024-09-21T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:22.144+0000 7fe8a5399640 1 -- 172.21.15.28:0/1589163457 wait complete. 2024-09-21T05:46:22.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:22 smithi028 ceph-mon[31227]: pgmap v305: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:22.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:22 smithi181 ceph-mon[34819]: pgmap v305: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:22 smithi129 ceph-mon[34885]: pgmap v305: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:22.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T05:46:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:23.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 -- 172.21.15.28:0/1772481036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 msgr2=0x7f0320110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 --2- 172.21.15.28:0/1772481036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f0320110620 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0318009ca0 tx=0x7f031802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 -- 172.21.15.28:0/1772481036 shutdown_connections 2024-09-21T05:46:23.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 --2- 172.21.15.28:0/1772481036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f0320110620 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 --2- 172.21.15.28:0/1772481036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 0x7f03201098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 --2- 172.21.15.28:0/1772481036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 0x7f0320103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 -- 172.21.15.28:0/1772481036 >> 172.21.15.28:0/1772481036 conn(0x7f032007b850 msgr2=0x7f032007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 -- 172.21.15.28:0/1772481036 shutdown_connections 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.687+0000 7f0324af7640 1 -- 172.21.15.28:0/1772481036 wait complete. 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.688+0000 7f0324af7640 1 Processor -- start 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.688+0000 7f0324af7640 1 -- start start 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 0x7f032019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 0x7f032019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113ab0 con 0x7f0320109df0 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113930 con 0x7f03201094a0 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0324af7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113c30 con 0x7f0320103b60 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44430/0 (socket says 172.21.15.28:44430) 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 -- 172.21.15.28:0/1579072792 learned_addr learned my addr 172.21.15.28:0/1579072792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031e59c640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 0x7f032019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f0317fff640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 0x7f032019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 -- 172.21.15.28:0/1579072792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 msgr2=0x7f032019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 0x7f032019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 -- 172.21.15.28:0/1579072792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 msgr2=0x7f032019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 0x7f032019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.689+0000 7f031ed9d640 1 -- 172.21.15.28:0/1579072792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03201a0290 con 0x7f0320109df0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f031ed9d640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f03180098d0 tx=0x7f03180317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f03177fe640 1 -- 172.21.15.28:0/1579072792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318003b70 con 0x7f0320109df0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f03177fe640 1 -- 172.21.15.28:0/1579072792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0318003d10 con 0x7f0320109df0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f03177fe640 1 -- 172.21.15.28:0/1579072792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031802f9d0 con 0x7f0320109df0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03201a0520 con 0x7f0320109df0 2024-09-21T05:46:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.690+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03201a0a10 con 0x7f0320109df0 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 Processor -- start 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f03177fe640 1 -- 172.21.15.28:0/1579072792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f031802fb70 con 0x7f0320109df0 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 -- start start 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0bde0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bde1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b0bd58c10 con 0x564b0ba120a0 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564b0bd48040 con 0x564b0ba1a280 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f032b06b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564b0bd4be50 con 0x564b0ba3af60 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.693+0000 7f031e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44446/0 (socket says 172.21.15.28:44446) 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 -- 172.21.15.28:0/438624559 learned_addr learned my addr 172.21.15.28:0/438624559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0317fff640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0bde0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0317fff640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bde1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f03177fe640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 0x7f030c0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 -- 172.21.15.28:0/438624559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 msgr2=0x564b0bde0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0bde0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 -- 172.21.15.28:0/438624559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 msgr2=0x564b0bde1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bde1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 -- 172.21.15.28:0/438624559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b0baa6540 con 0x564b0ba120a0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0317fff640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bde1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031ed9d640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 0x7f030c0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f031e59c640 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f031000ad00 tx=0x7f031000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360217dfaed34173 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f03177fe640 1 -- 172.21.15.28:0/1579072792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f03181085f0 con 0x7f0320109df0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0314ff9640 1 -- 172.21.15.28:0/438624559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031001d070 con 0x564b0ba120a0 2024-09-21T05:46:23.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0314ff9640 1 -- 172.21.15.28:0/438624559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0310018af0 con 0x564b0ba120a0 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.694+0000 7f0314ff9640 1 -- 172.21.15.28:0/438624559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310010b40 con 0x564b0ba120a0 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 -- 172.21.15.28:0/438624559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 msgr2=0x564b0bde1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f031ed9d640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 0x7f030c0b8aa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03180ce380 tx=0x7f03180095c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f031000ad00 tx=0x7f031000f790 comp rx=0 tx=0).stop 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 -- 172.21.15.28:0/438624559 shutdown_connections 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bde1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde1440 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 --2- 172.21.15.28:0/438624559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0bde0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 -- 172.21.15.28:0/438624559 >> 172.21.15.28:0/438624559 conn(0x564b0ba2f350 msgr2=0x564b0bddedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 -- 172.21.15.28:0/438624559 shutdown_connections 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.695+0000 7f032b06b740 1 -- 172.21.15.28:0/438624559 wait complete. 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 -- start start 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 Processor -- start 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba120a0 0x564b0bdec9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bdecf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b0bba95f0 con 0x564b0ba3af60 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564b0bce3010 con 0x564b0ba1a280 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdef6e0 con 0x564b0ba120a0 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f032b06b740 10 client.0 _ll_register_callbacks cb 0x564b0bd42340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44452/0 (socket says 172.21.15.28:44452) 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f0317fff640 1 -- 172.21.15.28:0/3012876989 learned_addr learned my addr 172.21.15.28:0/3012876989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.696+0000 7f0317fff640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba120a0 0x564b0bdec9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f031e59c640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bdecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 -- 172.21.15.28:0/3012876989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba120a0 msgr2=0x564b0bdec9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba120a0 0x564b0bdec9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 -- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 msgr2=0x564b0bdecf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bdecf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 -- 172.21.15.28:0/3012876989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b0bd10a00 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0317fff640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f030800f3c0 tx=0x7f03080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8005a0f86301be8c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0305ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800fe00 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0308009040 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0310018af0 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0308004020 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f032b06b740 10 client.15423 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564b0bdfb620 con 0x564b0ba3af60 2024-09-21T05:46:23.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.697+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0310018af0 con 0x564b0ba3af60 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f0308020040 con 0x564b0ba3af60 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 10 client.15423 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 10 client.15423 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 20 client.15423 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9f98e0 0x564b0bdff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 4 client.15423 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b0bd5ab60 con 0x564b0b9f98e0 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.698+0000 7f031e59c640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9f98e0 0x564b0bdff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.699+0000 7f031e59c640 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9f98e0 0x564b0bdff0e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1119b65319e5d03 server_cookie=202f7cee1aa88b1c in_seq=0 out_seq=0 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.699+0000 7f0305ffb640 10 client.15423 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:46:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.699+0000 7f0305ffb640 1 -- 172.21.15.28:0/3012876989 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564b0bd5ab60 con 0x564b0b9f98e0 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.699+0000 7f0305ffb640 10 client.15423 handle_command_reply: tid=0 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.699+0000 7f032b06b740 1 client.15423 shutdown 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 msgr2=0x564b0bdf89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f030800f3c0 tx=0x7f03080046c0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 shutdown_connections 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9f98e0 0x564b0bdff0e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba3af60 0x564b0bdf89d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0bdecf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 --2- 172.21.15.28:0/3012876989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba120a0 0x564b0bdec9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 >> 172.21.15.28:0/3012876989 conn(0x564b0ba2f350 msgr2=0x564b0bde4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f0305ffb640 0 client.15423 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:46:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 shutdown_connections 2024-09-21T05:46:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 1 -- 172.21.15.28:0/3012876989 wait complete. 2024-09-21T05:46:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.700+0000 7f032b06b740 20 client.15423 trim_cache size 0 max 16384 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 Processor -- start 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 -- start start 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba1a280 0x564b0bdf8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba3af60 0x564b0bdf8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdfd420 con 0x564b0ba120a0 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdfdaf0 con 0x564b0ba3af60 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f032b06b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdfe1c0 con 0x564b0ba1a280 2024-09-21T05:46:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44458/0 (socket says 172.21.15.28:44458) 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f0317fff640 1 -- 172.21.15.28:0/1493978694 learned_addr learned my addr 172.21.15.28:0/1493978694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.719+0000 7f031e59c640 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba1a280 0x564b0bdf8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba3af60 0x564b0bdf8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 -- 172.21.15.28:0/1493978694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba1a280 msgr2=0x564b0bdf8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba1a280 0x564b0bdf8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 -- 172.21.15.28:0/1493978694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba3af60 msgr2=0x564b0bdf8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba3af60 0x564b0bdf8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 -- 172.21.15.28:0/1493978694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b0bdd9080 con 0x564b0ba120a0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0317fff640 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0308006010 tx=0x7f0308011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36905c11a51389a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0307fff640 1 -- 172.21.15.28:0/1493978694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800fe00 con 0x564b0ba120a0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0307fff640 1 -- 172.21.15.28:0/1493978694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0308005290 con 0x564b0ba120a0 2024-09-21T05:46:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.720+0000 7f0307fff640 1 -- 172.21.15.28:0/1493978694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308004ca0 con 0x564b0ba120a0 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- 172.21.15.28:0/1493978694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 msgr2=0x564b0bde7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0308006010 tx=0x7f0308011cf0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- 172.21.15.28:0/1493978694 shutdown_connections 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba3af60 0x564b0bdf8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba1a280 0x564b0bdf8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 --2- 172.21.15.28:0/1493978694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0bde7020 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- 172.21.15.28:0/1493978694 >> 172.21.15.28:0/1493978694 conn(0x564b0ba2f350 msgr2=0x564b0bdeb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- 172.21.15.28:0/1493978694 shutdown_connections 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- 172.21.15.28:0/1493978694 wait complete. 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.721+0000 7f032b06b740 1 -- start start 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 Processor -- start 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0be01270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0be01960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdfef60 con 0x564b0ba120a0 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdfe890 con 0x564b0ba1a280 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564b0bdff630 con 0x564b0ba3af60 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f032b06b740 10 client.0 _ll_register_callbacks cb 0x7f0308011c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44462/0 (socket says 172.21.15.28:44462) 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 -- 172.21.15.28:0/3608350628 learned_addr learned my addr 172.21.15.28:0/3608350628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0be01960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f031e59c640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0be01270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 -- 172.21.15.28:0/3608350628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 msgr2=0x564b0be01960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0be01960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 -- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 msgr2=0x564b0be01270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0be01270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.722+0000 7f0317fff640 1 -- 172.21.15.28:0/3608350628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b0bded390 con 0x564b0ba120a0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0317fff640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0308011730 tx=0x7f0308004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ea8e4e469fb7c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308004c30 con 0x564b0ba120a0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f03080041c0 con 0x564b0ba120a0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0308005290 con 0x564b0ba120a0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0308043050 con 0x564b0ba120a0 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f032b06b740 10 client.15429 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564b0be6a5b0 con 0x564b0ba120a0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0308005290 con 0x564b0ba120a0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f030803c060 con 0x564b0ba120a0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f032b06b740 10 client.15429 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.723+0000 7f032b06b740 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f032b06b740 20 client.15429 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9cec90 0x564b0be6d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f032b06b740 4 client.15429 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "status"}) -- 0x564b0bd56610 con 0x564b0b9cec90 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f031e59c640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9cec90 0x564b0be6d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f031e59c640 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9cec90 0x564b0be6d8e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1108ee567ec25c7 server_cookie=f75c479db9565d8a in_seq=0 out_seq=0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f0305ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f0305ffb640 1 -- 172.21.15.28:0/3608350628 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564b0bd56610 con 0x564b0b9cec90 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f0305ffb640 10 client.15429 handle_command_reply: tid=0 2024-09-21T05:46:23.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.724+0000 7f032b06b740 1 client.15429 shutdown 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 msgr2=0x564b0be07d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0308011730 tx=0x7f0308004940 comp rx=0 tx=0).stop 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 shutdown_connections 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x564b0b9cec90 0x564b0be6d8e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564b0ba3af60 0x564b0be01960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564b0ba1a280 0x564b0be01270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 --2- 172.21.15.28:0/3608350628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b0ba120a0 0x564b0be07d20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 >> 172.21.15.28:0/3608350628 conn(0x564b0ba2f350 msgr2=0x564b0bdeacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f0305ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 shutdown_connections 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 1 -- 172.21.15.28:0/3608350628 wait complete. 2024-09-21T05:46:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.725+0000 7f032b06b740 20 client.15429 trim_cache size 0 max 16384 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24680, 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 95.555313820999999, 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:23.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:23.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:23.737 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 114.46610894600001 2024-09-21T05:46:23.737 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:23.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 msgr2=0x7f030c0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 0x7f030c0b8aa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03180ce380 tx=0x7f03180095c0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 msgr2=0x7f032019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f03180098d0 tx=0x7f03180317b0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 shutdown_connections 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f030c0b6590 0x7f030c0b8aa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0320109df0 0x7f032019fc20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201094a0 0x7f032019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 --2- 172.21.15.28:0/1579072792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0320103b60 0x7f032019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 >> 172.21.15.28:0/1579072792 conn(0x7f032007b850 msgr2=0x7f03201074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 shutdown_connections 2024-09-21T05:46:23.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:23.726+0000 7f0324af7640 1 -- 172.21.15.28:0/1579072792 wait complete. 2024-09-21T05:46:24.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T05:46:24.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:24 smithi181 ceph-mon[34819]: pgmap v306: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-21T05:46:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:24 smithi028 ceph-mon[31227]: pgmap v306: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-21T05:46:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:24 smithi129 ceph-mon[34885]: pgmap v306: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-21T05:46:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/487108279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c381094c0 msgr2=0x7f5c381098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/487108279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c381094c0 0x7f5c381098d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009900 tx=0x7f5c2002a410 comp rx=0 tx=0).stop 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/487108279 shutdown_connections 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/487108279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c38109e10 0x7f5c38110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/487108279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c381094c0 0x7f5c381098d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/487108279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38103b80 0x7f5c38103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/487108279 >> 172.21.15.28:0/487108279 conn(0x7f5c380ff610 msgr2=0x7f5c38101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/487108279 shutdown_connections 2024-09-21T05:46:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.262+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/487108279 wait complete. 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.263+0000 7f5c3c9b8640 1 Processor -- start 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.263+0000 7f5c3c9b8640 1 -- start start 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c381094c0 0x7f5c3819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38109e10 0x7f5c3819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c38113ad0 con 0x7f5c38103b80 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c38113950 con 0x7f5c38109e10 2024-09-21T05:46:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3c9b8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5c38113c50 con 0x7f5c381094c0 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36672/0 (socket says 172.21.15.28:36672) 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 -- 172.21.15.28:0/1211247798 learned_addr learned my addr 172.21.15.28:0/1211247798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c381094c0 0x7f5c3819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38109e10 0x7f5c3819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 -- 172.21.15.28:0/1211247798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c381094c0 msgr2=0x7f5c3819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c381094c0 0x7f5c3819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 -- 172.21.15.28:0/1211247798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38109e10 msgr2=0x7f5c3819fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38109e10 0x7f5c3819fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 -- 172.21.15.28:0/1211247798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c381a0350 con 0x7f5c38103b80 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c3659c640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000cce0 tx=0x7f5c3000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.264+0000 7f5c2f7fe640 1 -- 172.21.15.28:0/1211247798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000f650 con 0x7f5c38103b80 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.265+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c381a0640 con 0x7f5c38103b80 2024-09-21T05:46:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.265+0000 7f5c2f7fe640 1 -- 172.21.15.28:0/1211247798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c3000fca0 con 0x7f5c38103b80 2024-09-21T05:46:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.265+0000 7f5c2f7fe640 1 -- 172.21.15.28:0/1211247798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30018b40 con 0x7f5c38103b80 2024-09-21T05:46:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.265+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c380755f0 con 0x7f5c38103b80 2024-09-21T05:46:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.267+0000 7f5c42f2c740 1 Processor -- start 2024-09-21T05:46:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.267+0000 7f5c42f2c740 1 -- start start 2024-09-21T05:46:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.267+0000 7f5c2f7fe640 1 -- 172.21.15.28:0/1211247798 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f5c30018ce0 con 0x7f5c38103b80 2024-09-21T05:46:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1ce3280 0x5567b20aa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567b2021c10 con 0x5567b1cdb0a0 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567b2011040 con 0x5567b1d03f60 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567b2014e50 con 0x5567b1ce3280 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36684/0 (socket says 172.21.15.28:36684) 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 -- 172.21.15.28:0/2903875367 learned_addr learned my addr 172.21.15.28:0/2903875367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1ce3280 0x5567b20aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 -- 172.21.15.28:0/2903875367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1ce3280 msgr2=0x5567b20aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1ce3280 0x5567b20aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 -- 172.21.15.28:0/2903875367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 msgr2=0x5567b20a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 -- 172.21.15.28:0/2903875367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b1d6f540 con 0x5567b1cdb0a0 2024-09-21T05:46:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c2f7fe640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 0x7f5c240b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.268+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5c300edb80 tx=0x7f5c300edc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83628d7ce8e5edcc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.269+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 0x7f5c240b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.269+0000 7f5c2f7fe640 1 -- 172.21.15.28:0/1211247798 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5c300e7980 con 0x7f5c38103b80 2024-09-21T05:46:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.269+0000 7f5c2cff9640 1 -- 172.21.15.28:0/2903875367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c300aa060 con 0x5567b1cdb0a0 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.269+0000 7f5c2cff9640 1 -- 172.21.15.28:0/2903875367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c300ad520 con 0x5567b1cdb0a0 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.269+0000 7f5c2cff9640 1 -- 172.21.15.28:0/2903875367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c300ad180 con 0x5567b1cdb0a0 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2903875367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 msgr2=0x5567b20aa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5c300edb80 tx=0x7f5c300edc70 comp rx=0 tx=0).stop 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 0x7f5c240b8aa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800bca0 tx=0x7f5c2800b450 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2903875367 shutdown_connections 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1ce3280 0x5567b20aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20aa440 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2903875367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2903875367 >> 172.21.15.28:0/2903875367 conn(0x5567b1cf8350 msgr2=0x5567b20a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2903875367 shutdown_connections 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2903875367 wait complete. 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 -- start start 2024-09-21T05:46:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.270+0000 7f5c42f2c740 1 Processor -- start 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567b1e725f0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567b1fac010 con 0x5567b1ce3280 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567b20b86e0 con 0x5567b1d03f60 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c42f2c740 10 client.0 _ll_register_callbacks cb 0x5567b200b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36688/0 (socket says 172.21.15.28:36688) 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 -- 172.21.15.28:0/2591089786 learned_addr learned my addr 172.21.15.28:0/2591089786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:25.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 -- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 msgr2=0x5567b20c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 -- 172.21.15.28:0/2591089786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 msgr2=0x5567b20b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 -- 172.21.15.28:0/2591089786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b1fd9a00 con 0x5567b1cdb0a0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5c300ed1b0 tx=0x7f5c30007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f422422219cdda9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.271+0000 7f5c1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c300ad360 con 0x5567b1cdb0a0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5c30018400 con 0x5567b1cdb0a0 2024-09-21T05:46:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c300ad520 con 0x5567b1cdb0a0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c300ad7e0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 10 client.15441 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5567b20c4600 con 0x5567b1cdb0a0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c300ad520 con 0x5567b1cdb0a0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f5c300d7070 con 0x5567b1cdb0a0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 10 client.15441 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 10 client.15441 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 20 client.15441 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1cc28e0 0x5567b20c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 4 client.15441 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.272+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567b2023b60 con 0x5567b1cc28e0 2024-09-21T05:46:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.273+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1cc28e0 0x5567b20c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.273+0000 7f5c1dffb640 10 client.15441 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.273+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1cc28e0 0x5567b20c78a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=224a177a536eddbc server_cookie=41434cdf4dcb9f00 in_seq=0 out_seq=0 2024-09-21T05:46:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2591089786 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5567b2023b60 con 0x5567b1cc28e0 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c1dffb640 10 client.15441 handle_command_reply: tid=0 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 client.15441 shutdown 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 msgr2=0x5567b20b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5c300ed1b0 tx=0x7f5c30007f30 comp rx=0 tx=0).stop 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 shutdown_connections 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1cc28e0 0x5567b20c78a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c1dffb640 0 client.15441 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2591089786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20b5a20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 >> 172.21.15.28:0/2591089786 conn(0x5567b1cf8350 msgr2=0x5567b20ad0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 shutdown_connections 2024-09-21T05:46:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2591089786 wait complete. 2024-09-21T05:46:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.274+0000 7f5c42f2c740 20 client.15441 trim_cache size 0 max 16384 2024-09-21T05:46:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 Processor -- start 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 -- start start 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1cdb0a0 0x5567b20b0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c5be0 con 0x5567b1ce3280 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c62b0 con 0x5567b1d03f60 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c6980 con 0x5567b1cdb0a0 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36700/0 (socket says 172.21.15.28:36700) 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 -- 172.21.15.28:0/114155429 learned_addr learned my addr 172.21.15.28:0/114155429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c3659c640 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1cdb0a0 0x5567b20b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 -- 172.21.15.28:0/114155429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1cdb0a0 msgr2=0x5567b20b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1cdb0a0 0x5567b20b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 -- 172.21.15.28:0/114155429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 msgr2=0x5567b20c1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20c1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.293+0000 7f5c35d9b640 1 -- 172.21.15.28:0/114155429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b20a2080 con 0x5567b1ce3280 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009230 tx=0x7f5c20031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa45214a98077ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c1ffff640 1 -- 172.21.15.28:0/114155429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20035040 con 0x5567b1ce3280 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c1ffff640 1 -- 172.21.15.28:0/114155429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c20032930 con 0x5567b1ce3280 2024-09-21T05:46:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c1ffff640 1 -- 172.21.15.28:0/114155429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20004990 con 0x5567b1ce3280 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c42f2c740 1 -- 172.21.15.28:0/114155429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 msgr2=0x5567b20c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.294+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009230 tx=0x7f5c20031400 comp rx=0 tx=0).stop 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- 172.21.15.28:0/114155429 shutdown_connections 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1d03f60 0x5567b20c1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1ce3280 0x5567b20c1940 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/114155429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1cdb0a0 0x5567b20b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- 172.21.15.28:0/114155429 >> 172.21.15.28:0/114155429 conn(0x5567b1cf8350 msgr2=0x5567b20b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- 172.21.15.28:0/114155429 shutdown_connections 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- 172.21.15.28:0/114155429 wait complete. 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- start start 2024-09-21T05:46:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 Processor -- start 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c7720 con 0x5567b1cdb0a0 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c7050 con 0x5567b1ce3280 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c42f2c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567b20c7df0 con 0x5567b1d03f60 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.295+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c42f2c740 10 client.0 _ll_register_callbacks cb 0x7f5c28005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36708/0 (socket says 172.21.15.28:36708) 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 -- 172.21.15.28:0/2001091039 learned_addr learned my addr 172.21.15.28:0/2001091039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c36d9d640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 -- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 msgr2=0x5567b20c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 -- 172.21.15.28:0/2001091039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 msgr2=0x5567b20c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 -- 172.21.15.28:0/2001091039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b20ca4b0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c3659c640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5c300e53d0 tx=0x7f5c3000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82983c41a6d940ef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c300a98b0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5c300c3bc0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.296+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c20032930 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c300ad360 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 10 client.15447 fetch_fsmap learned FSMap version 26 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5567b20ca4b0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c20032930 con 0x5567b1cdb0a0 2024-09-21T05:46:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f5c300ac6e0 con 0x5567b1cdb0a0 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 10 client.15447 fetch_fsmap finished waiting for FSMap version 26 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 10 client.15447 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 20 client.15447 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1c97c90 0x5567b21361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 4 client.15447 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "status"}) -- 0x5567b201f610 con 0x5567b1c97c90 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.297+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1c97c90 0x5567b21361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c35d9b640 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1c97c90 0x5567b21361c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c926dbb707b6c21 server_cookie=fda76ba5fe81353 in_seq=0 out_seq=0 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c1dffb640 10 client.15447 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c1dffb640 1 -- 172.21.15.28:0/2001091039 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5567b201f610 con 0x5567b1c97c90 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c1dffb640 10 client.15447 handle_command_reply: tid=0 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 client.15447 shutdown 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 msgr2=0x5567b20d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5c300e53d0 tx=0x7f5c3000af40 comp rx=0 tx=0).stop 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 shutdown_connections 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5567b1c97c90 0x5567b21361c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567b1d03f60 0x5567b20c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567b1ce3280 0x5567b20c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 --2- 172.21.15.28:0/2001091039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567b1cdb0a0 0x5567b20d0330 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c1dffb640 0 client.15447 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 >> 172.21.15.28:0/2001091039 conn(0x5567b1cf8350 msgr2=0x5567b20a7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 shutdown_connections 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.298+0000 7f5c42f2c740 1 -- 172.21.15.28:0/2001091039 wait complete. 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c42f2c740 20 client.15447 trim_cache size 0 max 16384 2024-09-21T05:46:25.305 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "42500114-77db-11ef-baf5-efdc52797490", 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24688, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 105.175876656, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 26, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 73, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 71, 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 124.261373944 2024-09-21T05:46:25.306 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 msgr2=0x7f5c240b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 0x7f5c240b8aa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800bca0 tx=0x7f5c2800b450 comp rx=0 tx=0).stop 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 msgr2=0x7f5c3819a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000cce0 tx=0x7f5c3000a620 comp rx=0 tx=0).stop 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 shutdown_connections 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c38109e10 0x7f5c3819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c381094c0 0x7f5c3819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5c240b6590 0x7f5c240b8aa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 --2- 172.21.15.28:0/1211247798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c38103b80 0x7f5c3819a4a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 >> 172.21.15.28:0/1211247798 conn(0x7f5c380ff610 msgr2=0x7f5c38101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 shutdown_connections 2024-09-21T05:46:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:25.299+0000 7f5c3c9b8640 1 -- 172.21.15.28:0/1211247798 wait complete. 2024-09-21T05:46:26.099 INFO:teuthology.run_tasks:Running task kclient... 2024-09-21T05:46:26.125 INFO:tasks.kclient:Mounting kernel clients... 2024-09-21T05:46:26.125 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-21T05:46:26.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:26.126 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-21T05:46:26.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:26.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-21T05:46:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-21T05:46:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T05:46:26.216 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-09-21T05:46:26.216 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T05:46:26.216 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:26.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T05:46:26.515 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:26.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:26 smithi028 ceph-mon[31227]: pgmap v307: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:26.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:26 smithi181 ceph-mon[34819]: pgmap v307: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:26 smithi129 ceph-mon[34885]: pgmap v307: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T05:46:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 -- 172.21.15.28:0/2356760545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e241092d0 msgr2=0x7f1e241096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 --2- 172.21.15.28:0/2356760545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e241092d0 0x7f1e241096e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009900 tx=0x7f1e1802a410 comp rx=0 tx=0).stop 2024-09-21T05:46:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 -- 172.21.15.28:0/2356760545 shutdown_connections 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 --2- 172.21.15.28:0/2356760545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e24109c20 0x7f1e24110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 --2- 172.21.15.28:0/2356760545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e241092d0 0x7f1e241096e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 --2- 172.21.15.28:0/2356760545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24103990 0x7f1e24103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 -- 172.21.15.28:0/2356760545 >> 172.21.15.28:0/2356760545 conn(0x7f1e240ff440 msgr2=0x7f1e24101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 -- 172.21.15.28:0/2356760545 shutdown_connections 2024-09-21T05:46:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.980+0000 7f1e2939e640 1 -- 172.21.15.28:0/2356760545 wait complete. 2024-09-21T05:46:26.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.981+0000 7f1e2939e640 1 Processor -- start 2024-09-21T05:46:26.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.981+0000 7f1e2939e640 1 -- start start 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.981+0000 7f1e2939e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e2939e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e241092d0 0x7f1e2419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e2939e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24109c20 0x7f1e2419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e2939e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e241138e0 con 0x7f1e24103990 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e2939e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e24113760 con 0x7f1e24109c20 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e2939e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1e24113a60 con 0x7f1e241092d0 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36732/0 (socket says 172.21.15.28:36732) 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 -- 172.21.15.28:0/3595856447 learned_addr learned my addr 172.21.15.28:0/3595856447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e237fe640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e241092d0 0x7f1e2419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e28b9d640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24109c20 0x7f1e2419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 -- 172.21.15.28:0/3595856447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e241092d0 msgr2=0x7f1e2419a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e241092d0 0x7f1e2419a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 -- 172.21.15.28:0/3595856447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24109c20 msgr2=0x7f1e2419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24109c20 0x7f1e2419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 -- 172.21.15.28:0/3595856447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e241a0240 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e23fff640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400d950 tx=0x7f1e1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.982+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14017070 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.983+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e241a0530 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.983+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e14009c30 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.983+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1400f620 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.983+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e241a6c90 con 0x7f1e24103990 2024-09-21T05:46:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.984+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e24103da0 con 0x7f1e24103990 2024-09-21T05:46:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.985+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f1e14005030 con 0x7f1e24103990 2024-09-21T05:46:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.986+0000 7f1e217fa640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 0x7f1e0c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.986+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1e140e6fc0 con 0x7f1e24103990 2024-09-21T05:46:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.986+0000 7f1e237fe640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 0x7f1e0c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.987+0000 7f1e237fe640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 0x7f1e0c0b8a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009230 tx=0x7f1e18031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:26.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:26.989+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1e14009de0 con 0x7f1e24103990 2024-09-21T05:46:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.217+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1e2402e640 con 0x7f1e24103990 2024-09-21T05:46:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.217+0000 7f1e217fa640 1 -- 172.21.15.28:0/3595856447 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f1e140b1b30 con 0x7f1e24103990 2024-09-21T05:46:27.219 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.219+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 msgr2=0x7f1e0c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.219+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 0x7f1e0c0b8a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009230 tx=0x7f1e18031400 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.219+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 msgr2=0x7f1e2419a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.219+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400d950 tx=0x7f1e1400de10 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 shutdown_connections 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e24109c20 0x7f1e2419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e241092d0 0x7f1e2419a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e0c0b6560 0x7f1e0c0b8a70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 --2- 172.21.15.28:0/3595856447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e24103990 0x7f1e2419a2d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 >> 172.21.15.28:0/3595856447 conn(0x7f1e240ff440 msgr2=0x7f1e24101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 shutdown_connections 2024-09-21T05:46:27.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27.220+0000 7f1e2939e640 1 -- 172.21.15.28:0/3595856447 wait complete. 2024-09-21T05:46:27.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3595856447' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T05:46:27.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3595856447' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T05:46:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3595856447' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T05:46:27.882 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-21T05:46:27.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:27.883 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-21T05:46:27.901 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T05:46:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T05:46:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T05:46:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-21T05:46:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42145sec preferred_lft 42145sec 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-21T05:46:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-21T05:46:27.903 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T05:46:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-21T05:46:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-21T05:46:27.903 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T05:46:27.903 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T05:46:27.903 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-21T05:46:27.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-21T05:46:27.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-21T05:46:27.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-21T05:46:27.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T05:46:27.904 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-21T05:46:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:28.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:28.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:28.138 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T05:46:28.164 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-21T05:46:28.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:28.166 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-21T05:46:28.241 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-21T05:46:28.241 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T05:46:28.241 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T05:46:28.241 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T05:46:28.241 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T05:46:28.242 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-21T05:46:28.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:28.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:28.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:28.565 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-21T05:46:28.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:28.582 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-21T05:46:28.638 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T05:46:28.638 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-21T05:46:28.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T05:46:28.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-21T05:46:28.638 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-21T05:46:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:28.735 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:28 smithi028 ceph-mon[31227]: pgmap v308: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T05:46:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:28.758 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-21T05:46:28.758 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T05:46:28.758 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-21T05:46:28.759 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-21T05:46:28.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:28 smithi181 ceph-mon[34819]: pgmap v308: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T05:46:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:28 smithi129 ceph-mon[34885]: pgmap v308: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-21T05:46:28.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:28.907 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T05:46:28.907 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-21T05:46:28.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-21T05:46:28.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T05:46:28.907 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-21T05:46:28.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:28.986 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T05:46:28.986 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:29.015 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-21T05:46:29.015 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T05:46:29.016 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:29.071 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-21T05:46:29.071 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-21T05:46:29.071 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/7e06d2f3-dff5-4680-8244-da51f271037c 2024-09-21T05:46:29.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:29.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/7e06d2f3-dff5-4680-8244-da51f271037c /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-21T05:46:29.206 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-21T05:46:29.206 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-21T05:46:29.348 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-21T05:46:29.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-21T05:46:29.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.129:3300,172.21.15.181:3300" 2024-09-21T05:46:29.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@42500114-77db-11ef-baf5-efdc52797490.cephfs=/volumes/qa/sv_1/7e06d2f3-dff5-4680-8244-da51f271037c 2024-09-21T05:46:29.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.28:3300/172.21.15.129:3300/172.21.15.181:3300" will pass to kernel 2024-09-21T05:46:29.349 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-21T05:46:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-21T05:46:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-21T05:46:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-21T05:46:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-21T05:46:29.350 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-21T05:46:29.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:46:29.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:29.378 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T05:46:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T05:46:29.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:29.477 INFO:teuthology.orchestra.run.smithi028.stdout:client25177 2024-09-21T05:46:29.477 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:46:29.477 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T05:46:29.494 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T05:46:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T05:46:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:29.593 INFO:teuthology.orchestra.run.smithi028.stdout:client25177 2024-09-21T05:46:29.593 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-21T05:46:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T05:46:29.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:29.650 INFO:teuthology.orchestra.run.smithi028.stdout:client25177 2024-09-21T05:46:29.651 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/42500114-77db-11ef-baf5-efdc52797490.client25177/status') 2024-09-21T05:46:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T05:46:29.711 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25177 (3)172.21.15.28:0/540719452 2024-09-21T05:46:29.711 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-21T05:46:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-21T05:46:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-21T05:46:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 2.4125e-05 s, 2.9 MB/s 2024-09-21T05:46:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T05:46:29.719 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T05:46:29.730 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T05:46:29.730 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:46:29.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-09-21T05:46:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 -- 172.21.15.28:0/1747441589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 msgr2=0x7f07e0110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 --2- 172.21.15.28:0/1747441589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e0110280 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009ca0 tx=0x7f07d802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 -- 172.21.15.28:0/1747441589 shutdown_connections 2024-09-21T05:46:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 --2- 172.21.15.28:0/1747441589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e0110280 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 --2- 172.21.15.28:0/1747441589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e010d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 --2- 172.21.15.28:0/1747441589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e0104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 -- 172.21.15.28:0/1747441589 >> 172.21.15.28:0/1747441589 conn(0x7f07e00fe3e0 msgr2=0x7f07e0100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 -- 172.21.15.28:0/1747441589 shutdown_connections 2024-09-21T05:46:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.842+0000 7f07e60f2640 1 -- 172.21.15.28:0/1747441589 wait complete. 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 Processor -- start 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 -- start start 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07e01139f0 con 0x7f07e010dd90 2024-09-21T05:46:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07e0113870 con 0x7f07e0104930 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07e60f2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07e0113b70 con 0x7f07e0105280 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36752/0 (socket says 172.21.15.28:36752) 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.843+0000 7f07dffff640 1 -- 172.21.15.28:0/777679392 learned_addr learned my addr 172.21.15.28:0/777679392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07df7fe640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07d6ffd640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 -- 172.21.15.28:0/777679392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 msgr2=0x7f07e019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 -- 172.21.15.28:0/777679392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 msgr2=0x7f07e019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 -- 172.21.15.28:0/777679392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e01a0340 con 0x7f07e010dd90 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07df7fe640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07d6ffd640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dffff640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07d8005cc0 tx=0x7f07d802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8003bf0 con 0x7f07e010dd90 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07d8003d90 con 0x7f07e010dd90 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d802fbc0 con 0x7f07e010dd90 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e01a05d0 con 0x7f07e010dd90 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.844+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e01a6dc0 con 0x7f07e010dd90 2024-09-21T05:46:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.845+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e0104d40 con 0x7f07e010dd90 2024-09-21T05:46:29.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.848+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f07d802fd60 con 0x7f07e010dd90 2024-09-21T05:46:29.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.849+0000 7f07dd7fa640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 0x7f07cc0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.849+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f07d8108ba0 con 0x7f07e010dd90 2024-09-21T05:46:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.849+0000 7f07df7fe640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 0x7f07cc0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.849+0000 7f07df7fe640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 0x7f07cc0b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07d00058c0 tx=0x7f07d0005ec0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:29.852+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f07d80cf330 con 0x7f07e010dd90 2024-09-21T05:46:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.083+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f07e019b530 con 0x7f07e010dd90 2024-09-21T05:46:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.446+0000 7f07dd7fa640 1 -- 172.21.15.28:0/777679392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f07e019b530 con 0x7f07e010dd90 2024-09-21T05:46:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 msgr2=0x7f07cc0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 0x7f07cc0b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07d00058c0 tx=0x7f07d0005ec0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 msgr2=0x7f07e019fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07d8005cc0 tx=0x7f07d802f380 comp rx=0 tx=0).stop 2024-09-21T05:46:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 shutdown_connections 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07cc0b65e0 0x7f07cc0b8af0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07e010dd90 0x7f07e019fc40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07e0105280 0x7f07e019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 --2- 172.21.15.28:0/777679392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07e0104930 0x7f07e019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 >> 172.21.15.28:0/777679392 conn(0x7f07e00fe3e0 msgr2=0x7f07e0100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 shutdown_connections 2024-09-21T05:46:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.449+0000 7f07e60f2640 1 -- 172.21.15.28:0/777679392 wait complete. 2024-09-21T05:46:30.464 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-21T05:46:30.478 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-21T05:46:30.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T05:46:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 -- 172.21.15.28:0/2632434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 msgr2=0x7f42fc10f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 --2- 172.21.15.28:0/2632434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc10f430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009ca0 tx=0x7f42f402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 -- 172.21.15.28:0/2632434857 shutdown_connections 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 --2- 172.21.15.28:0/2632434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc10f430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 --2- 172.21.15.28:0/2632434857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc1082b0 0x7f42fc1086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 --2- 172.21.15.28:0/2632434857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc102980 0x7f42fc102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 -- 172.21.15.28:0/2632434857 >> 172.21.15.28:0/2632434857 conn(0x7f42fc0fe380 msgr2=0x7f42fc1007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 -- 172.21.15.28:0/2632434857 shutdown_connections 2024-09-21T05:46:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.631+0000 7f4303ba8640 1 -- 172.21.15.28:0/2632434857 wait complete. 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 Processor -- start 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 -- start start 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc102980 0x7f42fc19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc1082b0 0x7f42fc19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1128c0 con 0x7f42fc108c00 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc112740 con 0x7f42fc1082b0 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4303ba8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc112a40 con 0x7f42fc102980 2024-09-21T05:46:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4302145640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4302145640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36776/0 (socket says 172.21.15.28:36776) 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4302145640 1 -- 172.21.15.28:0/4228959395 learned_addr learned my addr 172.21.15.28:0/4228959395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.632+0000 7f4301944640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc102980 0x7f42fc19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4301143640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc1082b0 0x7f42fc19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 -- 172.21.15.28:0/4228959395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc102980 msgr2=0x7f42fc19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc102980 0x7f42fc19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 -- 172.21.15.28:0/4228959395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc1082b0 msgr2=0x7f42fc19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc1082b0 0x7f42fc19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 -- 172.21.15.28:0/4228959395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc1a0360 con 0x7f42fc108c00 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4302145640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f42f4005cc0 tx=0x7f42f402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4003bf0 con 0x7f42fc108c00 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42f4003d90 con 0x7f42fc108c00 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc1a05f0 con 0x7f42fc108c00 2024-09-21T05:46:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f402fbc0 con 0x7f42fc108c00 2024-09-21T05:46:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.633+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc1a6de0 con 0x7f42fc108c00 2024-09-21T05:46:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.634+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc103bf0 con 0x7f42fc108c00 2024-09-21T05:46:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.636+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f42f402fd60 con 0x7f42fc108c00 2024-09-21T05:46:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.636+0000 7f42f2ffd640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 0x7f42e80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.637+0000 7f4301944640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 0x7f42e80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.637+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f42f41089c0 con 0x7f42fc108c00 2024-09-21T05:46:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.637+0000 7f4301944640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 0x7f42e80b8af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f42f800bbe0 tx=0x7f42f800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.641+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f42f410ae20 con 0x7f42fc108c00 2024-09-21T05:46:30.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:30 smithi181 ceph-mon[34819]: pgmap v309: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T05:46:30.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:30 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T05:46:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:30 smithi028 ceph-mon[31227]: pgmap v309: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T05:46:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:30 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T05:46:30.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.831+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42fc1a7090 con 0x7f42fc108c00 2024-09-21T05:46:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.833+0000 7f42f2ffd640 1 -- 172.21.15.28:0/4228959395 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+23122 (secure 0 0 0) 0x7f42f40cf150 con 0x7f42fc108c00 2024-09-21T05:46:30.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:30.836 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:14.248651+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:46:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 msgr2=0x7f42e80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 0x7f42e80b8af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f42f800bbe0 tx=0x7f42f800b390 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 msgr2=0x7f42fc19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f42f4005cc0 tx=0x7f42f402f380 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 shutdown_connections 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42e80b65e0 0x7f42e80b8af0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42fc108c00 0x7f42fc19fc60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42fc1082b0 0x7f42fc19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 --2- 172.21.15.28:0/4228959395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc102980 0x7f42fc19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 >> 172.21.15.28:0/4228959395 conn(0x7f42fc0fe380 msgr2=0x7f42fc106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 shutdown_connections 2024-09-21T05:46:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.835+0000 7f4303ba8640 1 -- 172.21.15.28:0/4228959395 wait complete. 2024-09-21T05:46:30.849 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T05:40:11.086256+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-21T05:44:35.312867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-21T05:44:37.712198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T05:46:30.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-21T05:46:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:30 smithi129 ceph-mon[34885]: pgmap v309: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-21T05:46:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:30 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2013233691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8108300 msgr2=0x7fb9e8108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2013233691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8108300 0x7fb9e8108710 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009900 tx=0x7fb9dc02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2013233691 shutdown_connections 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2013233691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e810f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2013233691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8108300 0x7fb9e8108710 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2013233691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8102a20 0x7fb9e8102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2013233691 >> 172.21.15.28:0/2013233691 conn(0x7fb9e8079630 msgr2=0x7fb9e8079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2013233691 shutdown_connections 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.964+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2013233691 wait complete. 2024-09-21T05:46:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.965+0000 7fb9ee04d640 1 Processor -- start 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.965+0000 7fb9ee04d640 1 -- start start 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.965+0000 7fb9ee04d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.965+0000 7fb9ee04d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8108300 0x7fb9e819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9ee04d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9ee04d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8113a30 con 0x7fb9e8102a20 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9ee04d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e81138b0 con 0x7fb9e8108c50 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9ee04d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8113bb0 con 0x7fb9e8108300 2024-09-21T05:46:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 -- 172.21.15.28:0/2340336422 learned_addr learned my addr 172.21.15.28:0/2340336422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e7fff640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e6ffd640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8108300 0x7fb9e819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 -- 172.21.15.28:0/2340336422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8108300 msgr2=0x7fb9e819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8108300 0x7fb9e819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 -- 172.21.15.28:0/2340336422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 msgr2=0x7fb9e819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 -- 172.21.15.28:0/2340336422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e81a03c0 con 0x7fb9e8102a20 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e7fff640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e77fe640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800e9f0 tx=0x7fb9d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d800cd80 con 0x7fb9e8102a20 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e81a06b0 con 0x7fb9e8102a20 2024-09-21T05:46:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9d80054d0 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.966+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8010690 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.967+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81a6e40 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.968+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e8102e30 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.969+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7fb9d8004e90 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.970+0000 7fb9e4ff9640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 0x7fb9d00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.970+0000 7fb9e6ffd640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 0x7fb9d00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.970+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb9d80e7a60 con 0x7fb9e8102a20 2024-09-21T05:46:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.970+0000 7fb9e6ffd640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 0x7fb9d00b8a70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009230 tx=0x7fb9dc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:30.974+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb9d80ae190 con 0x7fb9e8102a20 2024-09-21T05:46:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.162+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9e810aa20 con 0x7fb9e8102a20 2024-09-21T05:46:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.163+0000 7fb9e4ff9640 1 -- 172.21.15.28:0/2340336422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7fb9d80b2570 con 0x7fb9e8102a20 2024-09-21T05:46:31.165 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-21T05:46:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 msgr2=0x7fb9d00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 0x7fb9d00b8a70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009230 tx=0x7fb9dc031400 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 msgr2=0x7fb9e819a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800e9f0 tx=0x7fb9d800edf0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 shutdown_connections 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e8108c50 0x7fb9e819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9e8108300 0x7fb9e819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9d00b6560 0x7fb9d00b8a70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 --2- 172.21.15.28:0/2340336422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9e8102a20 0x7fb9e819a490 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 >> 172.21.15.28:0/2340336422 conn(0x7fb9e8079630 msgr2=0x7fb9e8104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 shutdown_connections 2024-09-21T05:46:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.165+0000 7fb9ee04d640 1 -- 172.21.15.28:0/2340336422 wait complete. 2024-09-21T05:46:31.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.297+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/4277802698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0109d30 msgr2=0x7fc0f0110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.297+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/4277802698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0109d30 0x7fc0f0110560 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009ca0 tx=0x7fc0e802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/4277802698 shutdown_connections 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/4277802698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0109d30 0x7fc0f0110560 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/4277802698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f01093e0 0x7fc0f01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/4277802698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f0103ab0 0x7fc0f0103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/4277802698 >> 172.21.15.28:0/4277802698 conn(0x7fc0f00ff4d0 msgr2=0x7fc0f0101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/4277802698 shutdown_connections 2024-09-21T05:46:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/4277802698 wait complete. 2024-09-21T05:46:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 Processor -- start 2024-09-21T05:46:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.298+0000 7fc0f6b8b640 1 -- start start 2024-09-21T05:46:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f01093e0 0x7fc0f019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f0109d30 0x7fc0f019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f01139f0 con 0x7fc0f0103ab0 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0113870 con 0x7fc0f01093e0 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f6b8b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0113b70 con 0x7fc0f0109d30 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36818/0 (socket says 172.21.15.28:36818) 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 -- 172.21.15.28:0/3382270836 learned_addr learned my addr 172.21.15.28:0/3382270836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f5128640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f0109d30 0x7fc0f019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0effff640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f01093e0 0x7fc0f019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 -- 172.21.15.28:0/3382270836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f0109d30 msgr2=0x7fc0f019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f0109d30 0x7fc0f019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 -- 172.21.15.28:0/3382270836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f01093e0 msgr2=0x7fc0f019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f01093e0 0x7fc0f019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 -- 172.21.15.28:0/3382270836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f01a0300 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0f4927640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d950 tx=0x7fc0e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.299+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4017030 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.300+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f01a05f0 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.300+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0e4004020 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.300+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40047a0 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.300+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f01a6d80 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.301+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f0103ec0 con 0x7fc0f0103ab0 2024-09-21T05:46:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.302+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7fc0e4009a30 con 0x7fc0f0103ab0 2024-09-21T05:46:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.303+0000 7fc0edffb640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 0x7fc0e00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.303+0000 7fc0effff640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 0x7fc0e00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.303+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc0e40e7540 con 0x7fc0f0103ab0 2024-09-21T05:46:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.304+0000 7fc0effff640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 0x7fc0e00b8a70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0108e80 tx=0x7fc0dc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.306+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc0e40adc70 con 0x7fc0f0103ab0 2024-09-21T05:46:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.496+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0f002e640 con 0x7fc0f0103ab0 2024-09-21T05:46:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.496+0000 7fc0edffb640 1 -- 172.21.15.28:0/3382270836 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) ==== 104+0+11 (secure 0 0 0) 0x7fc0e40b2050 con 0x7fc0f0103ab0 2024-09-21T05:46:31.498 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 msgr2=0x7fc0e00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 0x7fc0e00b8a70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0108e80 tx=0x7fc0dc00b390 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 msgr2=0x7fc0f019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d950 tx=0x7fc0e400de10 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 shutdown_connections 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0f0109d30 0x7fc0f019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0f01093e0 0x7fc0f019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0e00b6560 0x7fc0e00b8a70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 --2- 172.21.15.28:0/3382270836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f0103ab0 0x7fc0f019a3d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 >> 172.21.15.28:0/3382270836 conn(0x7fc0f00ff4d0 msgr2=0x7fc0f01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 shutdown_connections 2024-09-21T05:46:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.499+0000 7fc0f6b8b640 1 -- 172.21.15.28:0/3382270836 wait complete. 2024-09-21T05:46:31.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-21T05:46:31.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 -- 172.21.15.28:0/1212175140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c102a20 msgr2=0x7ff98c102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 --2- 172.21.15.28:0/1212175140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c102a20 0x7ff98c102e30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff988009900 tx=0x7ff98802a410 comp rx=0 tx=0).stop 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 -- 172.21.15.28:0/1212175140 shutdown_connections 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 --2- 172.21.15.28:0/1212175140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 0x7ff98c10f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 --2- 172.21.15.28:0/1212175140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c108300 0x7ff98c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 --2- 172.21.15.28:0/1212175140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c102a20 0x7ff98c102e30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 -- 172.21.15.28:0/1212175140 >> 172.21.15.28:0/1212175140 conn(0x7ff98c079630 msgr2=0x7ff98c079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 -- 172.21.15.28:0/1212175140 shutdown_connections 2024-09-21T05:46:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.627+0000 7ff9940be640 1 -- 172.21.15.28:0/1212175140 wait complete. 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 Processor -- start 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 -- start start 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 0x7ff98c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 0x7ff98c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c113a30 con 0x7ff98c108300 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c1138b0 con 0x7ff98c108c50 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.628+0000 7ff9940be640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c113bb0 con 0x7ff98c102a20 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36836/0 (socket says 172.21.15.28:36836) 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 -- 172.21.15.28:0/1991302013 learned_addr learned my addr 172.21.15.28:0/1991302013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff99265b640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 0x7ff98c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991e5a640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 0x7ff98c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 -- 172.21.15.28:0/1991302013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 msgr2=0x7ff98c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 0x7ff98c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 -- 172.21.15.28:0/1991302013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 msgr2=0x7ff98c19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 0x7ff98c19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 -- 172.21.15.28:0/1991302013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c1a0310 con 0x7ff98c108300 2024-09-21T05:46:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991e5a640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 0x7ff98c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff991659640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00d950 tx=0x7ff97c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c017070 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97c009c30 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98c1a0600 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0047a0 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.629+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98c1a6e20 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.631+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98c102e30 con 0x7ff98c108300 2024-09-21T05:46:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.632+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7ff97c004020 con 0x7ff98c108300 2024-09-21T05:46:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.633+0000 7ff982ffd640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 0x7ff9780b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.634+0000 7ff991e5a640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 0x7ff9780b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.634+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff97c0e71b0 con 0x7ff98c108300 2024-09-21T05:46:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.634+0000 7ff991e5a640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 0x7ff9780b8a70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff9880096c0 tx=0x7ff988031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.638+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff97c020020 con 0x7ff98c108300 2024-09-21T05:46:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:31 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T05:46:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:31 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:46:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:31 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4228959395' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:31.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:31 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2340336422' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:46:31.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:31 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T05:46:31.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:31 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:46:31.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:31 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4228959395' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:31.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:31 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2340336422' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:46:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.824+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff98c10aa20 con 0x7ff98c108300 2024-09-21T05:46:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.825+0000 7ff982ffd640 1 -- 172.21.15.28:0/1991302013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) ==== 100+0+11 (secure 0 0 0) 0x7ff97c0ad940 con 0x7ff98c108300 2024-09-21T05:46:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 msgr2=0x7ff9780b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 0x7ff9780b8a70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff9880096c0 tx=0x7ff988031400 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 msgr2=0x7ff98c19aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00d950 tx=0x7ff97c00de10 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 shutdown_connections 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff98c108c50 0x7ff98c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff9780b6560 0x7ff9780b8a70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c108300 0x7ff98c19aa10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 --2- 172.21.15.28:0/1991302013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff98c102a20 0x7ff98c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 >> 172.21.15.28:0/1991302013 conn(0x7ff98c079630 msgr2=0x7ff98c1063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:31.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 shutdown_connections 2024-09-21T05:46:31.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:31.827+0000 7ff9940be640 1 -- 172.21.15.28:0/1991302013 wait complete. 2024-09-21T05:46:31.840 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-21T05:46:31.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:46:31.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:31 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/777679392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T05:46:31.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:31 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:46:31.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:31 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4228959395' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:31.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:31 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2340336422' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.570+0000 7f356e72c640 1 -- 172.21.15.28:0/1948133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 msgr2=0x7f35681087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.570+0000 7f356e72c640 1 --2- 172.21.15.28:0/1948133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f35681087e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f355c009900 tx=0x7f355c02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 -- 172.21.15.28:0/1948133842 shutdown_connections 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 --2- 172.21.15.28:0/1948133842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 0x7f356810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 --2- 172.21.15.28:0/1948133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f35681087e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 --2- 172.21.15.28:0/1948133842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 0x7f3568102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 -- 172.21.15.28:0/1948133842 >> 172.21.15.28:0/1948133842 conn(0x7f3568079750 msgr2=0x7f3568079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 -- 172.21.15.28:0/1948133842 shutdown_connections 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 -- 172.21.15.28:0/1948133842 wait complete. 2024-09-21T05:46:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 Processor -- start 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.571+0000 7f356e72c640 1 -- start start 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 0x7f356819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 0x7f356819fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113b50 con 0x7f35681083d0 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35681139d0 con 0x7f3568102a90 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356e72c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113cd0 con 0x7f3568108d20 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36848/0 (socket says 172.21.15.28:36848) 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 -- 172.21.15.28:0/2405153352 learned_addr learned my addr 172.21.15.28:0/2405153352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f3567fff640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 0x7f356819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f356ccc9640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 0x7f356819fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 -- 172.21.15.28:0/2405153352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 msgr2=0x7f356819fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 0x7f356819fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 -- 172.21.15.28:0/2405153352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 msgr2=0x7f356819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 0x7f356819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.572+0000 7f35677fe640 1 -- 172.21.15.28:0/2405153352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35681a0510 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f35677fe640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f355c031a90 tx=0x7f355c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c03f030 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f355c004490 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35681a07a0 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c035070 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.573+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35681a6f60 con 0x7f35681083d0 2024-09-21T05:46:32.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.574+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3568103d10 con 0x7f35681083d0 2024-09-21T05:46:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.576+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f355c042030 con 0x7f35681083d0 2024-09-21T05:46:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.577+0000 7f35657fa640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 0x7f35500b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.577+0000 7f3567fff640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 0x7f35500b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.577+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f355c107ef0 con 0x7f35681083d0 2024-09-21T05:46:32.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.578+0000 7f3567fff640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 0x7f35500b8a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3558009990 tx=0x7f3558009110 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:32.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.581+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f355c0ce620 con 0x7f35681083d0 2024-09-21T05:46:32.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:32 smithi028 ceph-mon[31227]: pgmap v310: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:32.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:32 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3382270836' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:32 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1991302013' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:32 smithi181 ceph-mon[34819]: pgmap v310: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:32 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3382270836' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:32 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1991302013' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.808+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f356810ab10 con 0x7f35681083d0 2024-09-21T05:46:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.808+0000 7f35657fa640 1 -- 172.21.15.28:0/2405153352 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7f355c0d2a00 con 0x7f35681083d0 2024-09-21T05:46:32.812 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:32.812 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:46:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 msgr2=0x7f35500b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 0x7f35500b8a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3558009990 tx=0x7f3558009110 comp rx=0 tx=0).stop 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 msgr2=0x7f356819ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f355c031a90 tx=0x7f355c004270 comp rx=0 tx=0).stop 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 shutdown_connections 2024-09-21T05:46:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3568108d20 0x7f356819fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f35500b6560 0x7f35500b8a70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35681083d0 0x7f356819ab40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 --2- 172.21.15.28:0/2405153352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3568102a90 0x7f356819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 >> 172.21.15.28:0/2405153352 conn(0x7f3568079750 msgr2=0x7f3568105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 shutdown_connections 2024-09-21T05:46:32.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:32.811+0000 7f356e72c640 1 -- 172.21.15.28:0/2405153352 wait complete. 2024-09-21T05:46:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:32 smithi129 ceph-mon[34885]: pgmap v310: 97 pgs: 97 active+clean; 647 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-21T05:46:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:32 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3382270836' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:46:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:32 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1991302013' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:46:33.608 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-21T05:46:33.608 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-21T05:46:33.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/115160500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 msgr2=0x7fbf381086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/115160500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf381086c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009900 tx=0x7fbf2c02a3f0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/115160500 shutdown_connections 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/115160500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3810f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/115160500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf381086c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/115160500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 0x7fbf38102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.725+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/115160500 >> 172.21.15.28:0/115160500 conn(0x7fbf380fe380 msgr2=0x7fbf381007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.726+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/115160500 shutdown_connections 2024-09-21T05:46:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.726+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/115160500 wait complete. 2024-09-21T05:46:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.726+0000 7fbf3e3eb640 1 Processor -- start 2024-09-21T05:46:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.726+0000 7fbf3e3eb640 1 -- start start 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 0x7fbf3819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf381139e0 con 0x7fbf381082b0 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38113860 con 0x7fbf38108c00 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3e3eb640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38113b60 con 0x7fbf38102980 2024-09-21T05:46:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36876/0 (socket says 172.21.15.28:36876) 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 -- 172.21.15.28:0/2422719359 learned_addr learned my addr 172.21.15.28:0/2422719359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf37fff640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 0x7fbf3819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3c988640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 -- 172.21.15.28:0/2422719359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 msgr2=0x7fbf3819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 0x7fbf3819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 -- 172.21.15.28:0/2422719359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 msgr2=0x7fbf3819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 -- 172.21.15.28:0/2422719359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf381a0240 con 0x7fbf381082b0 2024-09-21T05:46:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf3c988640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf377fe640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02a910 tx=0x7fbf2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.727+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c03f030 con 0x7fbf381082b0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.728+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf2c003aa0 con 0x7fbf381082b0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.728+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf381a04d0 con 0x7fbf381082b0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.728+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c035070 con 0x7fbf381082b0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.728+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf381a09c0 con 0x7fbf381082b0 2024-09-21T05:46:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.729+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf38103bf0 con 0x7fbf381082b0 2024-09-21T05:46:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.730+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7fbf2c042030 con 0x7fbf381082b0 2024-09-21T05:46:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.731+0000 7fbf357fa640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 0x7fbf200b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.731+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbf2c1080b0 con 0x7fbf381082b0 2024-09-21T05:46:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.731+0000 7fbf37fff640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 0x7fbf200b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.731+0000 7fbf37fff640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 0x7fbf200b8a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800abe0 tx=0x7fbf2800a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:33.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.734+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbf2c0ce7e0 con 0x7fbf381082b0 2024-09-21T05:46:33.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:33 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2405153352' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:33.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:33 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2405153352' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:33.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:33 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2405153352' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.919+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf38063800 con 0x7fbf200b6560 2024-09-21T05:46:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.923+0000 7fbf357fa640 1 -- 172.21.15.28:0/2422719359 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258879 (secure 0 0 0) 0x7fbf38063800 con 0x7fbf200b6560 2024-09-21T05:46:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:33.929 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:46:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 msgr2=0x7fbf200b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 0x7fbf200b8a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800abe0 tx=0x7fbf2800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 msgr2=0x7fbf3819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02a910 tx=0x7fbf2c004270 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 shutdown_connections 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf38108c00 0x7fbf3819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf200b6560 0x7fbf200b8a70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf381082b0 0x7fbf3819a940 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 --2- 172.21.15.28:0/2422719359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf38102980 0x7fbf3819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 >> 172.21.15.28:0/2422719359 conn(0x7fbf380fe380 msgr2=0x7fbf381058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 shutdown_connections 2024-09-21T05:46:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:33.927+0000 7fbf3e3eb640 1 -- 172.21.15.28:0/2422719359 wait complete. 2024-09-21T05:46:33.940 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-21T05:46:32.739991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668313,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1108,"num_read_kb":673,"num_write":376,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2953156,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394176,"kb_used_data":11648,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547648,"statfs":{"total":1151940427776,"available":1151536791552,"internally_reserved":0,"allocated":11927552,"data_stored":6766429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5751,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":77,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"3.3f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137687+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:46:13.137687+0000","last_peered":"2024-09-21T05:46:13.137687+0000","last_clean":"2024-09-21T05:46:13.137687+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:46:13.137687+0000","last_undegraded":"2024-09-21T05:46:13.137687+0000","last_fullsized":"2024-09-21T05:46:13.137687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.115376+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:46:13.115376+0000","last_peered":"2024-09-21T05:46:13.115376+0000","last_clean":"2024-09-21T05:46:13.115376+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:46:13.115376+0000","last_undegraded":"2024-09-21T05:46:13.115376+0000","last_fullsized":"2024-09-21T05:46:13.115376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133073+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:46:13.133073+0000","last_peered":"2024-09-21T05:46:13.133073+0000","last_clean":"2024-09-21T05:46:13.133073+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:46:13.133073+0000","last_undegraded":"2024-09-21T05:46:13.133073+0000","last_fullsized":"2024-09-21T05:46:13.133073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134375+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:46:13.134375+0000","last_peered":"2024-09-21T05:46:13.134375+0000","last_clean":"2024-09-21T05:46:13.134375+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:46:13.134375+0000","last_undegraded":"2024-09-21T05:46:13.134375+0000","last_fullsized":"2024-09-21T05:46:13.134375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129532+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:46:13.129532+0000","last_peered":"2024-09-21T05:46:13.129532+0000","last_clean":"2024-09-21T05:46:13.129532+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:46:13.129532+0000","last_undegraded":"2024-09-21T05:46:13.129532+0000","last_fullsized":"2024-09-21T05:46:13.129532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"73'6","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128199+0000","last_change":"2024-09-21T05:45:14.258964+0000","last_active":"2024-09-21T05:46:13.128199+0000","last_peered":"2024-09-21T05:46:13.128199+0000","last_clean":"2024-09-21T05:46:13.128199+0000","last_became_active":"2024-09-21T05:45:14.257864+0000","last_became_peered":"2024-09-21T05:45:14.257864+0000","last_unstale":"2024-09-21T05:46:13.128199+0000","last_undegraded":"2024-09-21T05:46:13.128199+0000","last_fullsized":"2024-09-21T05:46:13.128199+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.693452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133810+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:46:13.133810+0000","last_peered":"2024-09-21T05:46:13.133810+0000","last_clean":"2024-09-21T05:46:13.133810+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:46:13.133810+0000","last_undegraded":"2024-09-21T05:46:13.133810+0000","last_fullsized":"2024-09-21T05:46:13.133810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131962+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:46:13.131962+0000","last_peered":"2024-09-21T05:46:13.131962+0000","last_clean":"2024-09-21T05:46:13.131962+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:46:13.131962+0000","last_undegraded":"2024-09-21T05:46:13.131962+0000","last_fullsized":"2024-09-21T05:46:13.131962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138970+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:46:13.138970+0000","last_peered":"2024-09-21T05:46:13.138970+0000","last_clean":"2024-09-21T05:46:13.138970+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:46:13.138970+0000","last_undegraded":"2024-09-21T05:46:13.138970+0000","last_fullsized":"2024-09-21T05:46:13.138970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"73'16","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139172+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:46:13.139172+0000","last_peered":"2024-09-21T05:46:13.139172+0000","last_clean":"2024-09-21T05:46:13.139172+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:46:13.139172+0000","last_undegraded":"2024-09-21T05:46:13.139172+0000","last_fullsized":"2024-09-21T05:46:13.139172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135662+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:46:13.135662+0000","last_peered":"2024-09-21T05:46:13.135662+0000","last_clean":"2024-09-21T05:46:13.135662+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:46:13.135662+0000","last_undegraded":"2024-09-21T05:46:13.135662+0000","last_fullsized":"2024-09-21T05:46:13.135662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134240+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:46:13.134240+0000","last_peered":"2024-09-21T05:46:13.134240+0000","last_clean":"2024-09-21T05:46:13.134240+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:46:13.134240+0000","last_undegraded":"2024-09-21T05:46:13.134240+0000","last_fullsized":"2024-09-21T05:46:13.134240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129338+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:46:13.129338+0000","last_peered":"2024-09-21T05:46:13.129338+0000","last_clean":"2024-09-21T05:46:13.129338+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:46:13.129338+0000","last_undegraded":"2024-09-21T05:46:13.129338+0000","last_fullsized":"2024-09-21T05:46:13.129338+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122462+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:46:13.122462+0000","last_peered":"2024-09-21T05:46:13.122462+0000","last_clean":"2024-09-21T05:46:13.122462+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:46:13.122462+0000","last_undegraded":"2024-09-21T05:46:13.122462+0000","last_fullsized":"2024-09-21T05:46:13.122462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138684+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:46:13.138684+0000","last_peered":"2024-09-21T05:46:13.138684+0000","last_clean":"2024-09-21T05:46:13.138684+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:46:13.138684+0000","last_undegraded":"2024-09-21T05:46:13.138684+0000","last_fullsized":"2024-09-21T05:46:13.138684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131791+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:46:13.131791+0000","last_peered":"2024-09-21T05:46:13.131791+0000","last_clean":"2024-09-21T05:46:13.131791+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:46:13.131791+0000","last_undegraded":"2024-09-21T05:46:13.131791+0000","last_fullsized":"2024-09-21T05:46:13.131791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137966+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:46:13.137966+0000","last_peered":"2024-09-21T05:46:13.137966+0000","last_clean":"2024-09-21T05:46:13.137966+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:46:13.137966+0000","last_undegraded":"2024-09-21T05:46:13.137966+0000","last_fullsized":"2024-09-21T05:46:13.137966+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131930+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:46:13.131930+0000","last_peered":"2024-09-21T05:46:13.131930+0000","last_clean":"2024-09-21T05:46:13.131930+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:46:13.131930+0000","last_undegraded":"2024-09-21T05:46:13.131930+0000","last_fullsized":"2024-09-21T05:46:13.131930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129054+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:46:13.129054+0000","last_peered":"2024-09-21T05:46:13.129054+0000","last_clean":"2024-09-21T05:46:13.129054+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:46:13.129054+0000","last_undegraded":"2024-09-21T05:46:13.129054+0000","last_fullsized":"2024-09-21T05:46:13.129054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"73'8","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134973+0000","last_change":"2024-09-21T05:45:14.263089+0000","last_active":"2024-09-21T05:46:13.134973+0000","last_peered":"2024-09-21T05:46:13.134973+0000","last_clean":"2024-09-21T05:46:13.134973+0000","last_became_active":"2024-09-21T05:45:14.262626+0000","last_became_peered":"2024-09-21T05:45:14.262626+0000","last_unstale":"2024-09-21T05:46:13.134973+0000","last_undegraded":"2024-09-21T05:46:13.134973+0000","last_fullsized":"2024-09-21T05:46:13.134973+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:34.442511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130596+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:46:13.130596+0000","last_peered":"2024-09-21T05:46:13.130596+0000","last_clean":"2024-09-21T05:46:13.130596+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:46:13.130596+0000","last_undegraded":"2024-09-21T05:46:13.130596+0000","last_fullsized":"2024-09-21T05:46:13.130596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132727+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:46:13.132727+0000","last_peered":"2024-09-21T05:46:13.132727+0000","last_clean":"2024-09-21T05:46:13.132727+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:46:13.132727+0000","last_undegraded":"2024-09-21T05:46:13.132727+0000","last_fullsized":"2024-09-21T05:46:13.132727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130363+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:46:13.130363+0000","last_peered":"2024-09-21T05:46:13.130363+0000","last_clean":"2024-09-21T05:46:13.130363+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:46:13.130363+0000","last_undegraded":"2024-09-21T05:46:13.130363+0000","last_fullsized":"2024-09-21T05:46:13.130363+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130982+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:46:13.130982+0000","last_peered":"2024-09-21T05:46:13.130982+0000","last_clean":"2024-09-21T05:46:13.130982+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:46:13.130982+0000","last_undegraded":"2024-09-21T05:46:13.130982+0000","last_fullsized":"2024-09-21T05:46:13.130982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135513+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:46:13.135513+0000","last_peered":"2024-09-21T05:46:13.135513+0000","last_clean":"2024-09-21T05:46:13.135513+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:46:13.135513+0000","last_undegraded":"2024-09-21T05:46:13.135513+0000","last_fullsized":"2024-09-21T05:46:13.135513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132358+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:46:13.132358+0000","last_peered":"2024-09-21T05:46:13.132358+0000","last_clean":"2024-09-21T05:46:13.132358+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:46:13.132358+0000","last_undegraded":"2024-09-21T05:46:13.132358+0000","last_fullsized":"2024-09-21T05:46:13.132358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135025+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:46:13.135025+0000","last_peered":"2024-09-21T05:46:13.135025+0000","last_clean":"2024-09-21T05:46:13.135025+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:46:13.135025+0000","last_undegraded":"2024-09-21T05:46:13.135025+0000","last_fullsized":"2024-09-21T05:46:13.135025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134771+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:46:13.134771+0000","last_peered":"2024-09-21T05:46:13.134771+0000","last_clean":"2024-09-21T05:46:13.134771+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:46:13.134771+0000","last_undegraded":"2024-09-21T05:46:13.134771+0000","last_fullsized":"2024-09-21T05:46:13.134771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138706+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:46:13.138706+0000","last_peered":"2024-09-21T05:46:13.138706+0000","last_clean":"2024-09-21T05:46:13.138706+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:46:13.138706+0000","last_undegraded":"2024-09-21T05:46:13.138706+0000","last_fullsized":"2024-09-21T05:46:13.138706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127051+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:46:13.127051+0000","last_peered":"2024-09-21T05:46:13.127051+0000","last_clean":"2024-09-21T05:46:13.127051+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:46:13.127051+0000","last_undegraded":"2024-09-21T05:46:13.127051+0000","last_fullsized":"2024-09-21T05:46:13.127051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"73'12","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132243+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:46:13.132243+0000","last_peered":"2024-09-21T05:46:13.132243+0000","last_clean":"2024-09-21T05:46:13.132243+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:46:13.132243+0000","last_undegraded":"2024-09-21T05:46:13.132243+0000","last_fullsized":"2024-09-21T05:46:13.132243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139372+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:46:13.139372+0000","last_peered":"2024-09-21T05:46:13.139372+0000","last_clean":"2024-09-21T05:46:13.139372+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:46:13.139372+0000","last_undegraded":"2024-09-21T05:46:13.139372+0000","last_fullsized":"2024-09-21T05:46:13.139372+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122182+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:46:13.122182+0000","last_peered":"2024-09-21T05:46:13.122182+0000","last_clean":"2024-09-21T05:46:13.122182+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:46:13.122182+0000","last_undegraded":"2024-09-21T05:46:13.122182+0000","last_fullsized":"2024-09-21T05:46:13.122182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":144,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:31.374414+0000","last_change":"2024-09-21T05:45:14.267608+0000","last_active":"2024-09-21T05:46:31.374414+0000","last_peered":"2024-09-21T05:46:31.374414+0000","last_clean":"2024-09-21T05:46:31.374414+0000","last_became_active":"2024-09-21T05:45:14.260846+0000","last_became_peered":"2024-09-21T05:45:14.260846+0000","last_unstale":"2024-09-21T05:46:31.374414+0000","last_undegraded":"2024-09-21T05:46:31.374414+0000","last_fullsized":"2024-09-21T05:46:31.374414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:29.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134889+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:46:13.134889+0000","last_peered":"2024-09-21T05:46:13.134889+0000","last_clean":"2024-09-21T05:46:13.134889+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:46:13.134889+0000","last_undegraded":"2024-09-21T05:46:13.134889+0000","last_fullsized":"2024-09-21T05:46:13.134889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.275469+0000","last_change":"2024-09-21T05:45:14.275384+0000","last_active":"2024-09-21T05:45:14.275469+0000","last_peered":"2024-09-21T05:45:14.275469+0000","last_clean":"2024-09-21T05:45:14.275469+0000","last_became_active":"2024-09-21T05:45:14.265194+0000","last_became_peered":"2024-09-21T05:45:14.265194+0000","last_unstale":"2024-09-21T05:45:14.275469+0000","last_undegraded":"2024-09-21T05:45:14.275469+0000","last_fullsized":"2024-09-21T05:45:14.275469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:37:24.227108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252190+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:45:14.252190+0000","last_peered":"2024-09-21T05:45:14.252190+0000","last_clean":"2024-09-21T05:45:14.252190+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:45:14.252190+0000","last_undegraded":"2024-09-21T05:45:14.252190+0000","last_fullsized":"2024-09-21T05:45:14.252190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128150+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:46:13.128150+0000","last_peered":"2024-09-21T05:46:13.128150+0000","last_clean":"2024-09-21T05:46:13.128150+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:46:13.128150+0000","last_undegraded":"2024-09-21T05:46:13.128150+0000","last_fullsized":"2024-09-21T05:46:13.128150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.278537+0000","last_change":"2024-09-21T05:45:14.278120+0000","last_active":"2024-09-21T05:45:14.278537+0000","last_peered":"2024-09-21T05:45:14.278537+0000","last_clean":"2024-09-21T05:45:14.278537+0000","last_became_active":"2024-09-21T05:45:14.263713+0000","last_became_peered":"2024-09-21T05:45:14.263713+0000","last_unstale":"2024-09-21T05:45:14.278537+0000","last_undegraded":"2024-09-21T05:45:14.278537+0000","last_fullsized":"2024-09-21T05:45:14.278537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:16.259132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131917+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:46:13.131917+0000","last_peered":"2024-09-21T05:46:13.131917+0000","last_clean":"2024-09-21T05:46:13.131917+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:46:13.131917+0000","last_undegraded":"2024-09-21T05:46:13.131917+0000","last_fullsized":"2024-09-21T05:46:13.131917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"73'5","reported_seq":230,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.212715+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:46:32.212715+0000","last_peered":"2024-09-21T05:46:32.212715+0000","last_clean":"2024-09-21T05:46:32.212715+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:46:32.212715+0000","last_undegraded":"2024-09-21T05:46:32.212715+0000","last_fullsized":"2024-09-21T05:46:32.212715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":202,"num_read_kb":202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133943+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:46:13.133943+0000","last_peered":"2024-09-21T05:46:13.133943+0000","last_clean":"2024-09-21T05:46:13.133943+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:46:13.133943+0000","last_undegraded":"2024-09-21T05:46:13.133943+0000","last_fullsized":"2024-09-21T05:46:13.133943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.270877+0000","last_change":"2024-09-21T05:45:14.270775+0000","last_active":"2024-09-21T05:45:14.270877+0000","last_peered":"2024-09-21T05:45:14.270877+0000","last_clean":"2024-09-21T05:45:14.270877+0000","last_became_active":"2024-09-21T05:45:14.263657+0000","last_became_peered":"2024-09-21T05:45:14.263657+0000","last_unstale":"2024-09-21T05:45:14.270877+0000","last_undegraded":"2024-09-21T05:45:14.270877+0000","last_fullsized":"2024-09-21T05:45:14.270877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:27.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139619+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:46:13.139619+0000","last_peered":"2024-09-21T05:46:13.139619+0000","last_clean":"2024-09-21T05:46:13.139619+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:46:13.139619+0000","last_undegraded":"2024-09-21T05:46:13.139619+0000","last_fullsized":"2024-09-21T05:46:13.139619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253308+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:45:14.253308+0000","last_peered":"2024-09-21T05:45:14.253308+0000","last_clean":"2024-09-21T05:45:14.253308+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:45:14.253308+0000","last_undegraded":"2024-09-21T05:45:14.253308+0000","last_fullsized":"2024-09-21T05:45:14.253308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139494+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:46:13.139494+0000","last_peered":"2024-09-21T05:46:13.139494+0000","last_clean":"2024-09-21T05:46:13.139494+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:46:13.139494+0000","last_undegraded":"2024-09-21T05:46:13.139494+0000","last_fullsized":"2024-09-21T05:46:13.139494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"73'36","reported_seq":272,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.375781+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:46:32.375781+0000","last_peered":"2024-09-21T05:46:32.375781+0000","last_clean":"2024-09-21T05:46:32.375781+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:46:32.375781+0000","last_undegraded":"2024-09-21T05:46:32.375781+0000","last_fullsized":"2024-09-21T05:46:32.375781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70757,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":175,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.120849+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:46:13.120849+0000","last_peered":"2024-09-21T05:46:13.120849+0000","last_clean":"2024-09-21T05:46:13.120849+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:46:13.120849+0000","last_undegraded":"2024-09-21T05:46:13.120849+0000","last_fullsized":"2024-09-21T05:46:13.120849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256407+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:45:14.256407+0000","last_peered":"2024-09-21T05:45:14.256407+0000","last_clean":"2024-09-21T05:45:14.256407+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:45:14.256407+0000","last_undegraded":"2024-09-21T05:45:14.256407+0000","last_fullsized":"2024-09-21T05:45:14.256407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127982+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:46:13.127982+0000","last_peered":"2024-09-21T05:46:13.127982+0000","last_clean":"2024-09-21T05:46:13.127982+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:46:13.127982+0000","last_undegraded":"2024-09-21T05:46:13.127982+0000","last_fullsized":"2024-09-21T05:46:13.127982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253000+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:45:14.253000+0000","last_peered":"2024-09-21T05:45:14.253000+0000","last_clean":"2024-09-21T05:45:14.253000+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:45:14.253000+0000","last_undegraded":"2024-09-21T05:45:14.253000+0000","last_fullsized":"2024-09-21T05:45:14.253000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.141832+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:46:13.141832+0000","last_peered":"2024-09-21T05:46:13.141832+0000","last_clean":"2024-09-21T05:46:13.141832+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:46:13.141832+0000","last_undegraded":"2024-09-21T05:46:13.141832+0000","last_fullsized":"2024-09-21T05:46:13.141832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":127,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.205207+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:46:27.205207+0000","last_peered":"2024-09-21T05:46:27.205207+0000","last_clean":"2024-09-21T05:46:27.205207+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:46:27.205207+0000","last_undegraded":"2024-09-21T05:46:27.205207+0000","last_fullsized":"2024-09-21T05:46:27.205207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133477+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:46:13.133477+0000","last_peered":"2024-09-21T05:46:13.133477+0000","last_clean":"2024-09-21T05:46:13.133477+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:46:13.133477+0000","last_undegraded":"2024-09-21T05:46:13.133477+0000","last_fullsized":"2024-09-21T05:46:13.133477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":41,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258397+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:45:14.258397+0000","last_peered":"2024-09-21T05:45:14.258397+0000","last_clean":"2024-09-21T05:45:14.258397+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:45:14.258397+0000","last_undegraded":"2024-09-21T05:45:14.258397+0000","last_fullsized":"2024-09-21T05:45:14.258397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135098+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:46:13.135098+0000","last_peered":"2024-09-21T05:46:13.135098+0000","last_clean":"2024-09-21T05:46:13.135098+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:46:13.135098+0000","last_undegraded":"2024-09-21T05:46:13.135098+0000","last_fullsized":"2024-09-21T05:46:13.135098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259227+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:45:14.259227+0000","last_peered":"2024-09-21T05:45:14.259227+0000","last_clean":"2024-09-21T05:45:14.259227+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:45:14.259227+0000","last_undegraded":"2024-09-21T05:45:14.259227+0000","last_fullsized":"2024-09-21T05:45:14.259227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"73'5","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134830+0000","last_change":"2024-09-21T05:45:14.263088+0000","last_active":"2024-09-21T05:46:13.134830+0000","last_peered":"2024-09-21T05:46:13.134830+0000","last_clean":"2024-09-21T05:46:13.134830+0000","last_became_active":"2024-09-21T05:45:14.262643+0000","last_became_peered":"2024-09-21T05:45:14.262643+0000","last_unstale":"2024-09-21T05:46:13.134830+0000","last_undegraded":"2024-09-21T05:46:13.134830+0000","last_fullsized":"2024-09-21T05:46:13.134830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:05.991971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252502+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:45:14.252502+0000","last_peered":"2024-09-21T05:45:14.252502+0000","last_clean":"2024-09-21T05:45:14.252502+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:45:14.252502+0000","last_undegraded":"2024-09-21T05:45:14.252502+0000","last_fullsized":"2024-09-21T05:45:14.252502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130462+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:46:13.130462+0000","last_peered":"2024-09-21T05:46:13.130462+0000","last_clean":"2024-09-21T05:46:13.130462+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:46:13.130462+0000","last_undegraded":"2024-09-21T05:46:13.130462+0000","last_fullsized":"2024-09-21T05:46:13.130462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253487+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:45:14.253487+0000","last_peered":"2024-09-21T05:45:14.253487+0000","last_clean":"2024-09-21T05:45:14.253487+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:45:14.253487+0000","last_undegraded":"2024-09-21T05:45:14.253487+0000","last_fullsized":"2024-09-21T05:45:14.253487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133489+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:46:13.133489+0000","last_peered":"2024-09-21T05:46:13.133489+0000","last_clean":"2024-09-21T05:46:13.133489+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:46:13.133489+0000","last_undegraded":"2024-09-21T05:46:13.133489+0000","last_fullsized":"2024-09-21T05:46:13.133489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"73'7","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.210435+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:46:30.210435+0000","last_peered":"2024-09-21T05:46:30.210435+0000","last_clean":"2024-09-21T05:46:30.210435+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:46:30.210435+0000","last_undegraded":"2024-09-21T05:46:30.210435+0000","last_fullsized":"2024-09-21T05:46:30.210435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135958+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:46:13.135958+0000","last_peered":"2024-09-21T05:46:13.135958+0000","last_clean":"2024-09-21T05:46:13.135958+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:46:13.135958+0000","last_undegraded":"2024-09-21T05:46:13.135958+0000","last_fullsized":"2024-09-21T05:46:13.135958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257073+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:45:14.257073+0000","last_peered":"2024-09-21T05:45:14.257073+0000","last_clean":"2024-09-21T05:45:14.257073+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:45:14.257073+0000","last_undegraded":"2024-09-21T05:45:14.257073+0000","last_fullsized":"2024-09-21T05:45:14.257073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":32,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253471+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:45:14.253471+0000","last_peered":"2024-09-21T05:45:14.253471+0000","last_clean":"2024-09-21T05:45:14.253471+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:45:14.253471+0000","last_undegraded":"2024-09-21T05:45:14.253471+0000","last_fullsized":"2024-09-21T05:45:14.253471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130028+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:46:13.130028+0000","last_peered":"2024-09-21T05:46:13.130028+0000","last_clean":"2024-09-21T05:46:13.130028+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:46:13.130028+0000","last_undegraded":"2024-09-21T05:46:13.130028+0000","last_fullsized":"2024-09-21T05:46:13.130028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256020+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:45:14.256020+0000","last_peered":"2024-09-21T05:45:14.256020+0000","last_clean":"2024-09-21T05:45:14.256020+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:45:14.256020+0000","last_undegraded":"2024-09-21T05:45:14.256020+0000","last_fullsized":"2024-09-21T05:45:14.256020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133409+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:46:13.133409+0000","last_peered":"2024-09-21T05:46:13.133409+0000","last_clean":"2024-09-21T05:46:13.133409+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:46:13.133409+0000","last_undegraded":"2024-09-21T05:46:13.133409+0000","last_fullsized":"2024-09-21T05:46:13.133409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":129,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.209048+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:46:30.209048+0000","last_peered":"2024-09-21T05:46:30.209048+0000","last_clean":"2024-09-21T05:46:30.209048+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:46:30.209048+0000","last_undegraded":"2024-09-21T05:46:30.209048+0000","last_fullsized":"2024-09-21T05:46:30.209048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134643+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:46:13.134643+0000","last_peered":"2024-09-21T05:46:13.134643+0000","last_clean":"2024-09-21T05:46:13.134643+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:46:13.134643+0000","last_undegraded":"2024-09-21T05:46:13.134643+0000","last_fullsized":"2024-09-21T05:46:13.134643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":128,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:29.208025+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:46:29.208025+0000","last_peered":"2024-09-21T05:46:29.208025+0000","last_clean":"2024-09-21T05:46:29.208025+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:46:29.208025+0000","last_undegraded":"2024-09-21T05:46:29.208025+0000","last_fullsized":"2024-09-21T05:46:29.208025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138161+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:46:13.138161+0000","last_peered":"2024-09-21T05:46:13.138161+0000","last_clean":"2024-09-21T05:46:13.138161+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:46:13.138161+0000","last_undegraded":"2024-09-21T05:46:13.138161+0000","last_fullsized":"2024-09-21T05:46:13.138161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255492+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:45:14.255492+0000","last_peered":"2024-09-21T05:45:14.255492+0000","last_clean":"2024-09-21T05:45:14.255492+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:45:14.255492+0000","last_undegraded":"2024-09-21T05:45:14.255492+0000","last_fullsized":"2024-09-21T05:45:14.255492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.124320+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:46:13.124320+0000","last_peered":"2024-09-21T05:46:13.124320+0000","last_clean":"2024-09-21T05:46:13.124320+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:46:13.124320+0000","last_undegraded":"2024-09-21T05:46:13.124320+0000","last_fullsized":"2024-09-21T05:46:13.124320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.269448+0000","last_change":"2024-09-21T05:45:14.268855+0000","last_active":"2024-09-21T05:45:14.269448+0000","last_peered":"2024-09-21T05:45:14.269448+0000","last_clean":"2024-09-21T05:45:14.269448+0000","last_became_active":"2024-09-21T05:45:14.261558+0000","last_became_peered":"2024-09-21T05:45:14.261558+0000","last_unstale":"2024-09-21T05:45:14.269448+0000","last_undegraded":"2024-09-21T05:45:14.269448+0000","last_fullsized":"2024-09-21T05:45:14.269448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:57.978428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130304+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:46:13.130304+0000","last_peered":"2024-09-21T05:46:13.130304+0000","last_clean":"2024-09-21T05:46:13.130304+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:46:13.130304+0000","last_undegraded":"2024-09-21T05:46:13.130304+0000","last_fullsized":"2024-09-21T05:46:13.130304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"73'15","reported_seq":38,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255700+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:14.255700+0000","last_peered":"2024-09-21T05:45:14.255700+0000","last_clean":"2024-09-21T05:45:14.255700+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:14.255700+0000","last_undegraded":"2024-09-21T05:45:14.255700+0000","last_fullsized":"2024-09-21T05:45:14.255700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"73'5","reported_seq":17,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134660+0000","last_change":"2024-09-21T05:45:14.261447+0000","last_active":"2024-09-21T05:46:13.134660+0000","last_peered":"2024-09-21T05:46:13.134660+0000","last_clean":"2024-09-21T05:46:13.134660+0000","last_became_active":"2024-09-21T05:45:14.261072+0000","last_became_peered":"2024-09-21T05:45:14.261072+0000","last_unstale":"2024-09-21T05:46:13.134660+0000","last_undegraded":"2024-09-21T05:46:13.134660+0000","last_fullsized":"2024-09-21T05:46:13.134660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:46.148923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254328+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:45:14.254328+0000","last_peered":"2024-09-21T05:45:14.254328+0000","last_clean":"2024-09-21T05:45:14.254328+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:45:14.254328+0000","last_undegraded":"2024-09-21T05:45:14.254328+0000","last_fullsized":"2024-09-21T05:45:14.254328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139836+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:46:13.139836+0000","last_peered":"2024-09-21T05:46:13.139836+0000","last_clean":"2024-09-21T05:46:13.139836+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:46:13.139836+0000","last_undegraded":"2024-09-21T05:46:13.139836+0000","last_fullsized":"2024-09-21T05:46:13.139836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262195+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:45:14.262195+0000","last_peered":"2024-09-21T05:45:14.262195+0000","last_clean":"2024-09-21T05:45:14.262195+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:45:14.262195+0000","last_undegraded":"2024-09-21T05:45:14.262195+0000","last_fullsized":"2024-09-21T05:45:14.262195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.094131+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:46:13.094131+0000","last_peered":"2024-09-21T05:46:13.094131+0000","last_clean":"2024-09-21T05:46:13.094131+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:46:13.094131+0000","last_undegraded":"2024-09-21T05:46:13.094131+0000","last_fullsized":"2024-09-21T05:46:13.094131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321694+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:45:14.321694+0000","last_peered":"2024-09-21T05:45:14.321694+0000","last_clean":"2024-09-21T05:45:14.321694+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:45:14.321694+0000","last_undegraded":"2024-09-21T05:45:14.321694+0000","last_fullsized":"2024-09-21T05:45:14.321694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139062+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:46:13.139062+0000","last_peered":"2024-09-21T05:46:13.139062+0000","last_clean":"2024-09-21T05:46:13.139062+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:46:13.139062+0000","last_undegraded":"2024-09-21T05:46:13.139062+0000","last_fullsized":"2024-09-21T05:46:13.139062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256606+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:45:14.256606+0000","last_peered":"2024-09-21T05:45:14.256606+0000","last_clean":"2024-09-21T05:45:14.256606+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:45:14.256606+0000","last_undegraded":"2024-09-21T05:45:14.256606+0000","last_fullsized":"2024-09-21T05:45:14.256606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.126834+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:46:13.126834+0000","last_peered":"2024-09-21T05:46:13.126834+0000","last_clean":"2024-09-21T05:46:13.126834+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:46:13.126834+0000","last_undegraded":"2024-09-21T05:46:13.126834+0000","last_fullsized":"2024-09-21T05:46:13.126834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258860+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:45:14.258860+0000","last_peered":"2024-09-21T05:45:14.258860+0000","last_clean":"2024-09-21T05:45:14.258860+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:45:14.258860+0000","last_undegraded":"2024-09-21T05:45:14.258860+0000","last_fullsized":"2024-09-21T05:45:14.258860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135626+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:46:13.135626+0000","last_peered":"2024-09-21T05:46:13.135626+0000","last_clean":"2024-09-21T05:46:13.135626+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:46:13.135626+0000","last_undegraded":"2024-09-21T05:46:13.135626+0000","last_fullsized":"2024-09-21T05:46:13.135626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"73'15","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132962+0000","last_change":"2024-09-21T05:45:14.263694+0000","last_active":"2024-09-21T05:46:13.132962+0000","last_peered":"2024-09-21T05:46:13.132962+0000","last_clean":"2024-09-21T05:46:13.132962+0000","last_became_active":"2024-09-21T05:45:14.263394+0000","last_became_peered":"2024-09-21T05:45:14.263394+0000","last_unstale":"2024-09-21T05:46:13.132962+0000","last_undegraded":"2024-09-21T05:46:13.132962+0000","last_fullsized":"2024-09-21T05:46:13.132962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:59.214553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260360+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:45:14.260360+0000","last_peered":"2024-09-21T05:45:14.260360+0000","last_clean":"2024-09-21T05:45:14.260360+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:45:14.260360+0000","last_undegraded":"2024-09-21T05:45:14.260360+0000","last_fullsized":"2024-09-21T05:45:14.260360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134106+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:46:13.134106+0000","last_peered":"2024-09-21T05:46:13.134106+0000","last_clean":"2024-09-21T05:46:13.134106+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:46:13.134106+0000","last_undegraded":"2024-09-21T05:46:13.134106+0000","last_fullsized":"2024-09-21T05:46:13.134106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":36,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258852+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:45:14.258852+0000","last_peered":"2024-09-21T05:45:14.258852+0000","last_clean":"2024-09-21T05:45:14.258852+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:45:14.258852+0000","last_undegraded":"2024-09-21T05:45:14.258852+0000","last_fullsized":"2024-09-21T05:45:14.258852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133083+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:46:13.133083+0000","last_peered":"2024-09-21T05:46:13.133083+0000","last_clean":"2024-09-21T05:46:13.133083+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:46:13.133083+0000","last_undegraded":"2024-09-21T05:46:13.133083+0000","last_fullsized":"2024-09-21T05:46:13.133083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322148+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:45:14.322148+0000","last_peered":"2024-09-21T05:45:14.322148+0000","last_clean":"2024-09-21T05:45:14.322148+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:45:14.322148+0000","last_undegraded":"2024-09-21T05:45:14.322148+0000","last_fullsized":"2024-09-21T05:45:14.322148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130113+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:46:13.130113+0000","last_peered":"2024-09-21T05:46:13.130113+0000","last_clean":"2024-09-21T05:46:13.130113+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:46:13.130113+0000","last_undegraded":"2024-09-21T05:46:13.130113+0000","last_fullsized":"2024-09-21T05:46:13.130113+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.204331+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:46:27.204331+0000","last_peered":"2024-09-21T05:46:27.204331+0000","last_clean":"2024-09-21T05:46:27.204331+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:46:27.204331+0000","last_undegraded":"2024-09-21T05:46:27.204331+0000","last_fullsized":"2024-09-21T05:46:27.204331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77711,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1027,"num_read_kb":604,"num_write":251,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1316,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":233133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874276,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36144,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709008,"statfs":{"total":95995035648,"available":95958024192,"internally_reserved":0,"allocated":819200,"data_stored":400116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037801,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36828,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708324,"statfs":{"total":95995035648,"available":95957323776,"internally_reserved":0,"allocated":1519616,"data_stored":1060676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201325,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36156,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708996,"statfs":{"total":95995035648,"available":95958011904,"internally_reserved":0,"allocated":831488,"data_stored":398820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364850,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":819200,"data_stored":399086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561079,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1425408,"data_stored":991325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":815104,"data_stored":398537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1433600,"data_stored":990421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":823296,"data_stored":399020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":831488,"data_stored":398362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":892928,"data_stored":464894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":880640,"data_stored":464764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":835584,"data_stored":400408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":72639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":72493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:46:33.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- 172.21.15.28:0/2205774478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4108c00 msgr2=0x7f45d410f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 --2- 172.21.15.28:0/2205774478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4108c00 0x7f45d410f430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009ca0 tx=0x7f45cc02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- 172.21.15.28:0/2205774478 shutdown_connections 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 --2- 172.21.15.28:0/2205774478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4108c00 0x7f45d410f430 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 --2- 172.21.15.28:0/2205774478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d41086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 --2- 172.21.15.28:0/2205774478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4102980 0x7f45d4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- 172.21.15.28:0/2205774478 >> 172.21.15.28:0/2205774478 conn(0x7f45d40fe380 msgr2=0x7f45d41007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- 172.21.15.28:0/2205774478 shutdown_connections 2024-09-21T05:46:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- 172.21.15.28:0/2205774478 wait complete. 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 Processor -- start 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.096+0000 7f45dbf13640 1 -- start start 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 0x7f45d419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41128c0 con 0x7f45d4102980 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4112740 con 0x7f45d41082b0 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45dbf13640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4112a40 con 0x7f45d4108c00 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36884/0 (socket says 172.21.15.28:36884) 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 -- 172.21.15.28:0/4019993322 learned_addr learned my addr 172.21.15.28:0/4019993322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45da4b0640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 0x7f45d419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d94ae640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 -- 172.21.15.28:0/4019993322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 msgr2=0x7f45d419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 0x7f45d419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 -- 172.21.15.28:0/4019993322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 msgr2=0x7f45d419a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d419a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 -- 172.21.15.28:0/4019993322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d41a02a0 con 0x7f45d4102980 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45da4b0640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 0x7f45d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d94ae640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.097+0000 7f45d9caf640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f45d000d950 tx=0x7f45d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.098+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0017030 con 0x7f45d4102980 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.098+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45d0004020 con 0x7f45d4102980 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.098+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41a0590 con 0x7f45d4102980 2024-09-21T05:46:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.098+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d00047a0 con 0x7f45d4102980 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.098+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a6d20 con 0x7f45d4102980 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.099+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d4102d90 con 0x7f45d4102980 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.101+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f45d0009a30 con 0x7f45d4102980 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.101+0000 7f45caffd640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 0x7f45c00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.102+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f45d00e7660 con 0x7f45d4102980 2024-09-21T05:46:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.102+0000 7f45d94ae640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 0x7f45c00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.102+0000 7f45d94ae640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 0x7f45c00b8a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f45d406bd20 tx=0x7f45c40094b0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.108+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f45d00addf0 con 0x7f45d4102980 2024-09-21T05:46:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.295+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45d4063800 con 0x7f45c00b6560 2024-09-21T05:46:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.300+0000 7f45caffd640 1 -- 172.21.15.28:0/4019993322 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258879 (secure 0 0 0) 0x7f45d4063800 con 0x7f45c00b6560 2024-09-21T05:46:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 msgr2=0x7f45c00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 0x7f45c00b8a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f45d406bd20 tx=0x7f45c40094b0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 msgr2=0x7f45d419a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f45d000d950 tx=0x7f45d000de10 comp rx=0 tx=0).stop 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 shutdown_connections 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45d4108c00 0x7f45d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45d41082b0 0x7f45d419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f45c00b6560 0x7f45c00b8a70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 --2- 172.21.15.28:0/4019993322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4102980 0x7f45d419a370 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 >> 172.21.15.28:0/4019993322 conn(0x7f45d40fe380 msgr2=0x7f45d406aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 shutdown_connections 2024-09-21T05:46:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.303+0000 7f45dbf13640 1 -- 172.21.15.28:0/4019993322 wait complete. 2024-09-21T05:46:34.317 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-21T05:46:32.739991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668313,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1108,"num_read_kb":673,"num_write":376,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2953156,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394176,"kb_used_data":11648,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547648,"statfs":{"total":1151940427776,"available":1151536791552,"internally_reserved":0,"allocated":11927552,"data_stored":6766429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5751,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":77,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"3.3f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137687+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:46:13.137687+0000","last_peered":"2024-09-21T05:46:13.137687+0000","last_clean":"2024-09-21T05:46:13.137687+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:46:13.137687+0000","last_undegraded":"2024-09-21T05:46:13.137687+0000","last_fullsized":"2024-09-21T05:46:13.137687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.115376+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:46:13.115376+0000","last_peered":"2024-09-21T05:46:13.115376+0000","last_clean":"2024-09-21T05:46:13.115376+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:46:13.115376+0000","last_undegraded":"2024-09-21T05:46:13.115376+0000","last_fullsized":"2024-09-21T05:46:13.115376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133073+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:46:13.133073+0000","last_peered":"2024-09-21T05:46:13.133073+0000","last_clean":"2024-09-21T05:46:13.133073+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:46:13.133073+0000","last_undegraded":"2024-09-21T05:46:13.133073+0000","last_fullsized":"2024-09-21T05:46:13.133073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134375+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:46:13.134375+0000","last_peered":"2024-09-21T05:46:13.134375+0000","last_clean":"2024-09-21T05:46:13.134375+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:46:13.134375+0000","last_undegraded":"2024-09-21T05:46:13.134375+0000","last_fullsized":"2024-09-21T05:46:13.134375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129532+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:46:13.129532+0000","last_peered":"2024-09-21T05:46:13.129532+0000","last_clean":"2024-09-21T05:46:13.129532+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:46:13.129532+0000","last_undegraded":"2024-09-21T05:46:13.129532+0000","last_fullsized":"2024-09-21T05:46:13.129532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"73'6","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128199+0000","last_change":"2024-09-21T05:45:14.258964+0000","last_active":"2024-09-21T05:46:13.128199+0000","last_peered":"2024-09-21T05:46:13.128199+0000","last_clean":"2024-09-21T05:46:13.128199+0000","last_became_active":"2024-09-21T05:45:14.257864+0000","last_became_peered":"2024-09-21T05:45:14.257864+0000","last_unstale":"2024-09-21T05:46:13.128199+0000","last_undegraded":"2024-09-21T05:46:13.128199+0000","last_fullsized":"2024-09-21T05:46:13.128199+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.693452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133810+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:46:13.133810+0000","last_peered":"2024-09-21T05:46:13.133810+0000","last_clean":"2024-09-21T05:46:13.133810+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:46:13.133810+0000","last_undegraded":"2024-09-21T05:46:13.133810+0000","last_fullsized":"2024-09-21T05:46:13.133810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131962+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:46:13.131962+0000","last_peered":"2024-09-21T05:46:13.131962+0000","last_clean":"2024-09-21T05:46:13.131962+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:46:13.131962+0000","last_undegraded":"2024-09-21T05:46:13.131962+0000","last_fullsized":"2024-09-21T05:46:13.131962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138970+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:46:13.138970+0000","last_peered":"2024-09-21T05:46:13.138970+0000","last_clean":"2024-09-21T05:46:13.138970+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:46:13.138970+0000","last_undegraded":"2024-09-21T05:46:13.138970+0000","last_fullsized":"2024-09-21T05:46:13.138970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"73'16","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139172+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:46:13.139172+0000","last_peered":"2024-09-21T05:46:13.139172+0000","last_clean":"2024-09-21T05:46:13.139172+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:46:13.139172+0000","last_undegraded":"2024-09-21T05:46:13.139172+0000","last_fullsized":"2024-09-21T05:46:13.139172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135662+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:46:13.135662+0000","last_peered":"2024-09-21T05:46:13.135662+0000","last_clean":"2024-09-21T05:46:13.135662+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:46:13.135662+0000","last_undegraded":"2024-09-21T05:46:13.135662+0000","last_fullsized":"2024-09-21T05:46:13.135662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134240+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:46:13.134240+0000","last_peered":"2024-09-21T05:46:13.134240+0000","last_clean":"2024-09-21T05:46:13.134240+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:46:13.134240+0000","last_undegraded":"2024-09-21T05:46:13.134240+0000","last_fullsized":"2024-09-21T05:46:13.134240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129338+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:46:13.129338+0000","last_peered":"2024-09-21T05:46:13.129338+0000","last_clean":"2024-09-21T05:46:13.129338+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:46:13.129338+0000","last_undegraded":"2024-09-21T05:46:13.129338+0000","last_fullsized":"2024-09-21T05:46:13.129338+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122462+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:46:13.122462+0000","last_peered":"2024-09-21T05:46:13.122462+0000","last_clean":"2024-09-21T05:46:13.122462+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:46:13.122462+0000","last_undegraded":"2024-09-21T05:46:13.122462+0000","last_fullsized":"2024-09-21T05:46:13.122462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138684+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:46:13.138684+0000","last_peered":"2024-09-21T05:46:13.138684+0000","last_clean":"2024-09-21T05:46:13.138684+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:46:13.138684+0000","last_undegraded":"2024-09-21T05:46:13.138684+0000","last_fullsized":"2024-09-21T05:46:13.138684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131791+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:46:13.131791+0000","last_peered":"2024-09-21T05:46:13.131791+0000","last_clean":"2024-09-21T05:46:13.131791+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:46:13.131791+0000","last_undegraded":"2024-09-21T05:46:13.131791+0000","last_fullsized":"2024-09-21T05:46:13.131791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137966+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:46:13.137966+0000","last_peered":"2024-09-21T05:46:13.137966+0000","last_clean":"2024-09-21T05:46:13.137966+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:46:13.137966+0000","last_undegraded":"2024-09-21T05:46:13.137966+0000","last_fullsized":"2024-09-21T05:46:13.137966+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131930+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:46:13.131930+0000","last_peered":"2024-09-21T05:46:13.131930+0000","last_clean":"2024-09-21T05:46:13.131930+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:46:13.131930+0000","last_undegraded":"2024-09-21T05:46:13.131930+0000","last_fullsized":"2024-09-21T05:46:13.131930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129054+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:46:13.129054+0000","last_peered":"2024-09-21T05:46:13.129054+0000","last_clean":"2024-09-21T05:46:13.129054+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:46:13.129054+0000","last_undegraded":"2024-09-21T05:46:13.129054+0000","last_fullsized":"2024-09-21T05:46:13.129054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"73'8","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134973+0000","last_change":"2024-09-21T05:45:14.263089+0000","last_active":"2024-09-21T05:46:13.134973+0000","last_peered":"2024-09-21T05:46:13.134973+0000","last_clean":"2024-09-21T05:46:13.134973+0000","last_became_active":"2024-09-21T05:45:14.262626+0000","last_became_peered":"2024-09-21T05:45:14.262626+0000","last_unstale":"2024-09-21T05:46:13.134973+0000","last_undegraded":"2024-09-21T05:46:13.134973+0000","last_fullsized":"2024-09-21T05:46:13.134973+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:34.442511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130596+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:46:13.130596+0000","last_peered":"2024-09-21T05:46:13.130596+0000","last_clean":"2024-09-21T05:46:13.130596+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:46:13.130596+0000","last_undegraded":"2024-09-21T05:46:13.130596+0000","last_fullsized":"2024-09-21T05:46:13.130596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132727+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:46:13.132727+0000","last_peered":"2024-09-21T05:46:13.132727+0000","last_clean":"2024-09-21T05:46:13.132727+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:46:13.132727+0000","last_undegraded":"2024-09-21T05:46:13.132727+0000","last_fullsized":"2024-09-21T05:46:13.132727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130363+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:46:13.130363+0000","last_peered":"2024-09-21T05:46:13.130363+0000","last_clean":"2024-09-21T05:46:13.130363+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:46:13.130363+0000","last_undegraded":"2024-09-21T05:46:13.130363+0000","last_fullsized":"2024-09-21T05:46:13.130363+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130982+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:46:13.130982+0000","last_peered":"2024-09-21T05:46:13.130982+0000","last_clean":"2024-09-21T05:46:13.130982+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:46:13.130982+0000","last_undegraded":"2024-09-21T05:46:13.130982+0000","last_fullsized":"2024-09-21T05:46:13.130982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135513+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:46:13.135513+0000","last_peered":"2024-09-21T05:46:13.135513+0000","last_clean":"2024-09-21T05:46:13.135513+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:46:13.135513+0000","last_undegraded":"2024-09-21T05:46:13.135513+0000","last_fullsized":"2024-09-21T05:46:13.135513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132358+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:46:13.132358+0000","last_peered":"2024-09-21T05:46:13.132358+0000","last_clean":"2024-09-21T05:46:13.132358+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:46:13.132358+0000","last_undegraded":"2024-09-21T05:46:13.132358+0000","last_fullsized":"2024-09-21T05:46:13.132358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135025+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:46:13.135025+0000","last_peered":"2024-09-21T05:46:13.135025+0000","last_clean":"2024-09-21T05:46:13.135025+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:46:13.135025+0000","last_undegraded":"2024-09-21T05:46:13.135025+0000","last_fullsized":"2024-09-21T05:46:13.135025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134771+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:46:13.134771+0000","last_peered":"2024-09-21T05:46:13.134771+0000","last_clean":"2024-09-21T05:46:13.134771+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:46:13.134771+0000","last_undegraded":"2024-09-21T05:46:13.134771+0000","last_fullsized":"2024-09-21T05:46:13.134771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138706+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:46:13.138706+0000","last_peered":"2024-09-21T05:46:13.138706+0000","last_clean":"2024-09-21T05:46:13.138706+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:46:13.138706+0000","last_undegraded":"2024-09-21T05:46:13.138706+0000","last_fullsized":"2024-09-21T05:46:13.138706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127051+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:46:13.127051+0000","last_peered":"2024-09-21T05:46:13.127051+0000","last_clean":"2024-09-21T05:46:13.127051+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:46:13.127051+0000","last_undegraded":"2024-09-21T05:46:13.127051+0000","last_fullsized":"2024-09-21T05:46:13.127051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"73'12","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132243+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:46:13.132243+0000","last_peered":"2024-09-21T05:46:13.132243+0000","last_clean":"2024-09-21T05:46:13.132243+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:46:13.132243+0000","last_undegraded":"2024-09-21T05:46:13.132243+0000","last_fullsized":"2024-09-21T05:46:13.132243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139372+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:46:13.139372+0000","last_peered":"2024-09-21T05:46:13.139372+0000","last_clean":"2024-09-21T05:46:13.139372+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:46:13.139372+0000","last_undegraded":"2024-09-21T05:46:13.139372+0000","last_fullsized":"2024-09-21T05:46:13.139372+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122182+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:46:13.122182+0000","last_peered":"2024-09-21T05:46:13.122182+0000","last_clean":"2024-09-21T05:46:13.122182+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:46:13.122182+0000","last_undegraded":"2024-09-21T05:46:13.122182+0000","last_fullsized":"2024-09-21T05:46:13.122182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":144,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:31.374414+0000","last_change":"2024-09-21T05:45:14.267608+0000","last_active":"2024-09-21T05:46:31.374414+0000","last_peered":"2024-09-21T05:46:31.374414+0000","last_clean":"2024-09-21T05:46:31.374414+0000","last_became_active":"2024-09-21T05:45:14.260846+0000","last_became_peered":"2024-09-21T05:45:14.260846+0000","last_unstale":"2024-09-21T05:46:31.374414+0000","last_undegraded":"2024-09-21T05:46:31.374414+0000","last_fullsized":"2024-09-21T05:46:31.374414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:29.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134889+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:46:13.134889+0000","last_peered":"2024-09-21T05:46:13.134889+0000","last_clean":"2024-09-21T05:46:13.134889+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:46:13.134889+0000","last_undegraded":"2024-09-21T05:46:13.134889+0000","last_fullsized":"2024-09-21T05:46:13.134889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.275469+0000","last_change":"2024-09-21T05:45:14.275384+0000","last_active":"2024-09-21T05:45:14.275469+0000","last_peered":"2024-09-21T05:45:14.275469+0000","last_clean":"2024-09-21T05:45:14.275469+0000","last_became_active":"2024-09-21T05:45:14.265194+0000","last_became_peered":"2024-09-21T05:45:14.265194+0000","last_unstale":"2024-09-21T05:45:14.275469+0000","last_undegraded":"2024-09-21T05:45:14.275469+0000","last_fullsized":"2024-09-21T05:45:14.275469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:37:24.227108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252190+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:45:14.252190+0000","last_peered":"2024-09-21T05:45:14.252190+0000","last_clean":"2024-09-21T05:45:14.252190+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:45:14.252190+0000","last_undegraded":"2024-09-21T05:45:14.252190+0000","last_fullsized":"2024-09-21T05:45:14.252190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128150+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:46:13.128150+0000","last_peered":"2024-09-21T05:46:13.128150+0000","last_clean":"2024-09-21T05:46:13.128150+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:46:13.128150+0000","last_undegraded":"2024-09-21T05:46:13.128150+0000","last_fullsized":"2024-09-21T05:46:13.128150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.278537+0000","last_change":"2024-09-21T05:45:14.278120+0000","last_active":"2024-09-21T05:45:14.278537+0000","last_peered":"2024-09-21T05:45:14.278537+0000","last_clean":"2024-09-21T05:45:14.278537+0000","last_became_active":"2024-09-21T05:45:14.263713+0000","last_became_peered":"2024-09-21T05:45:14.263713+0000","last_unstale":"2024-09-21T05:45:14.278537+0000","last_undegraded":"2024-09-21T05:45:14.278537+0000","last_fullsized":"2024-09-21T05:45:14.278537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:16.259132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131917+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:46:13.131917+0000","last_peered":"2024-09-21T05:46:13.131917+0000","last_clean":"2024-09-21T05:46:13.131917+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:46:13.131917+0000","last_undegraded":"2024-09-21T05:46:13.131917+0000","last_fullsized":"2024-09-21T05:46:13.131917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"73'5","reported_seq":230,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.212715+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:46:32.212715+0000","last_peered":"2024-09-21T05:46:32.212715+0000","last_clean":"2024-09-21T05:46:32.212715+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:46:32.212715+0000","last_undegraded":"2024-09-21T05:46:32.212715+0000","last_fullsized":"2024-09-21T05:46:32.212715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":202,"num_read_kb":202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133943+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:46:13.133943+0000","last_peered":"2024-09-21T05:46:13.133943+0000","last_clean":"2024-09-21T05:46:13.133943+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:46:13.133943+0000","last_undegraded":"2024-09-21T05:46:13.133943+0000","last_fullsized":"2024-09-21T05:46:13.133943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.270877+0000","last_change":"2024-09-21T05:45:14.270775+0000","last_active":"2024-09-21T05:45:14.270877+0000","last_peered":"2024-09-21T05:45:14.270877+0000","last_clean":"2024-09-21T05:45:14.270877+0000","last_became_active":"2024-09-21T05:45:14.263657+0000","last_became_peered":"2024-09-21T05:45:14.263657+0000","last_unstale":"2024-09-21T05:45:14.270877+0000","last_undegraded":"2024-09-21T05:45:14.270877+0000","last_fullsized":"2024-09-21T05:45:14.270877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:27.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139619+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:46:13.139619+0000","last_peered":"2024-09-21T05:46:13.139619+0000","last_clean":"2024-09-21T05:46:13.139619+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:46:13.139619+0000","last_undegraded":"2024-09-21T05:46:13.139619+0000","last_fullsized":"2024-09-21T05:46:13.139619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253308+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:45:14.253308+0000","last_peered":"2024-09-21T05:45:14.253308+0000","last_clean":"2024-09-21T05:45:14.253308+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:45:14.253308+0000","last_undegraded":"2024-09-21T05:45:14.253308+0000","last_fullsized":"2024-09-21T05:45:14.253308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139494+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:46:13.139494+0000","last_peered":"2024-09-21T05:46:13.139494+0000","last_clean":"2024-09-21T05:46:13.139494+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:46:13.139494+0000","last_undegraded":"2024-09-21T05:46:13.139494+0000","last_fullsized":"2024-09-21T05:46:13.139494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"73'36","reported_seq":272,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.375781+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:46:32.375781+0000","last_peered":"2024-09-21T05:46:32.375781+0000","last_clean":"2024-09-21T05:46:32.375781+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:46:32.375781+0000","last_undegraded":"2024-09-21T05:46:32.375781+0000","last_fullsized":"2024-09-21T05:46:32.375781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70757,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":175,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.120849+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:46:13.120849+0000","last_peered":"2024-09-21T05:46:13.120849+0000","last_clean":"2024-09-21T05:46:13.120849+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:46:13.120849+0000","last_undegraded":"2024-09-21T05:46:13.120849+0000","last_fullsized":"2024-09-21T05:46:13.120849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256407+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:45:14.256407+0000","last_peered":"2024-09-21T05:45:14.256407+0000","last_clean":"2024-09-21T05:45:14.256407+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:45:14.256407+0000","last_undegraded":"2024-09-21T05:45:14.256407+0000","last_fullsized":"2024-09-21T05:45:14.256407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127982+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:46:13.127982+0000","last_peered":"2024-09-21T05:46:13.127982+0000","last_clean":"2024-09-21T05:46:13.127982+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:46:13.127982+0000","last_undegraded":"2024-09-21T05:46:13.127982+0000","last_fullsized":"2024-09-21T05:46:13.127982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253000+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:45:14.253000+0000","last_peered":"2024-09-21T05:45:14.253000+0000","last_clean":"2024-09-21T05:45:14.253000+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:45:14.253000+0000","last_undegraded":"2024-09-21T05:45:14.253000+0000","last_fullsized":"2024-09-21T05:45:14.253000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.141832+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:46:13.141832+0000","last_peered":"2024-09-21T05:46:13.141832+0000","last_clean":"2024-09-21T05:46:13.141832+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:46:13.141832+0000","last_undegraded":"2024-09-21T05:46:13.141832+0000","last_fullsized":"2024-09-21T05:46:13.141832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":127,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.205207+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:46:27.205207+0000","last_peered":"2024-09-21T05:46:27.205207+0000","last_clean":"2024-09-21T05:46:27.205207+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:46:27.205207+0000","last_undegraded":"2024-09-21T05:46:27.205207+0000","last_fullsized":"2024-09-21T05:46:27.205207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133477+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:46:13.133477+0000","last_peered":"2024-09-21T05:46:13.133477+0000","last_clean":"2024-09-21T05:46:13.133477+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:46:13.133477+0000","last_undegraded":"2024-09-21T05:46:13.133477+0000","last_fullsized":"2024-09-21T05:46:13.133477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":41,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258397+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:45:14.258397+0000","last_peered":"2024-09-21T05:45:14.258397+0000","last_clean":"2024-09-21T05:45:14.258397+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:45:14.258397+0000","last_undegraded":"2024-09-21T05:45:14.258397+0000","last_fullsized":"2024-09-21T05:45:14.258397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135098+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:46:13.135098+0000","last_peered":"2024-09-21T05:46:13.135098+0000","last_clean":"2024-09-21T05:46:13.135098+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:46:13.135098+0000","last_undegraded":"2024-09-21T05:46:13.135098+0000","last_fullsized":"2024-09-21T05:46:13.135098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259227+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:45:14.259227+0000","last_peered":"2024-09-21T05:45:14.259227+0000","last_clean":"2024-09-21T05:45:14.259227+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:45:14.259227+0000","last_undegraded":"2024-09-21T05:45:14.259227+0000","last_fullsized":"2024-09-21T05:45:14.259227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"73'5","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134830+0000","last_change":"2024-09-21T05:45:14.263088+0000","last_active":"2024-09-21T05:46:13.134830+0000","last_peered":"2024-09-21T05:46:13.134830+0000","last_clean":"2024-09-21T05:46:13.134830+0000","last_became_active":"2024-09-21T05:45:14.262643+0000","last_became_peered":"2024-09-21T05:45:14.262643+0000","last_unstale":"2024-09-21T05:46:13.134830+0000","last_undegraded":"2024-09-21T05:46:13.134830+0000","last_fullsized":"2024-09-21T05:46:13.134830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:05.991971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252502+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:45:14.252502+0000","last_peered":"2024-09-21T05:45:14.252502+0000","last_clean":"2024-09-21T05:45:14.252502+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:45:14.252502+0000","last_undegraded":"2024-09-21T05:45:14.252502+0000","last_fullsized":"2024-09-21T05:45:14.252502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130462+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:46:13.130462+0000","last_peered":"2024-09-21T05:46:13.130462+0000","last_clean":"2024-09-21T05:46:13.130462+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:46:13.130462+0000","last_undegraded":"2024-09-21T05:46:13.130462+0000","last_fullsized":"2024-09-21T05:46:13.130462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253487+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:45:14.253487+0000","last_peered":"2024-09-21T05:45:14.253487+0000","last_clean":"2024-09-21T05:45:14.253487+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:45:14.253487+0000","last_undegraded":"2024-09-21T05:45:14.253487+0000","last_fullsized":"2024-09-21T05:45:14.253487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133489+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:46:13.133489+0000","last_peered":"2024-09-21T05:46:13.133489+0000","last_clean":"2024-09-21T05:46:13.133489+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:46:13.133489+0000","last_undegraded":"2024-09-21T05:46:13.133489+0000","last_fullsized":"2024-09-21T05:46:13.133489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"73'7","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.210435+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:46:30.210435+0000","last_peered":"2024-09-21T05:46:30.210435+0000","last_clean":"2024-09-21T05:46:30.210435+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:46:30.210435+0000","last_undegraded":"2024-09-21T05:46:30.210435+0000","last_fullsized":"2024-09-21T05:46:30.210435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135958+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:46:13.135958+0000","last_peered":"2024-09-21T05:46:13.135958+0000","last_clean":"2024-09-21T05:46:13.135958+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:46:13.135958+0000","last_undegraded":"2024-09-21T05:46:13.135958+0000","last_fullsized":"2024-09-21T05:46:13.135958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257073+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:45:14.257073+0000","last_peered":"2024-09-21T05:45:14.257073+0000","last_clean":"2024-09-21T05:45:14.257073+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:45:14.257073+0000","last_undegraded":"2024-09-21T05:45:14.257073+0000","last_fullsized":"2024-09-21T05:45:14.257073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":32,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253471+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:45:14.253471+0000","last_peered":"2024-09-21T05:45:14.253471+0000","last_clean":"2024-09-21T05:45:14.253471+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:45:14.253471+0000","last_undegraded":"2024-09-21T05:45:14.253471+0000","last_fullsized":"2024-09-21T05:45:14.253471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130028+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:46:13.130028+0000","last_peered":"2024-09-21T05:46:13.130028+0000","last_clean":"2024-09-21T05:46:13.130028+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:46:13.130028+0000","last_undegraded":"2024-09-21T05:46:13.130028+0000","last_fullsized":"2024-09-21T05:46:13.130028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256020+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:45:14.256020+0000","last_peered":"2024-09-21T05:45:14.256020+0000","last_clean":"2024-09-21T05:45:14.256020+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:45:14.256020+0000","last_undegraded":"2024-09-21T05:45:14.256020+0000","last_fullsized":"2024-09-21T05:45:14.256020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133409+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:46:13.133409+0000","last_peered":"2024-09-21T05:46:13.133409+0000","last_clean":"2024-09-21T05:46:13.133409+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:46:13.133409+0000","last_undegraded":"2024-09-21T05:46:13.133409+0000","last_fullsized":"2024-09-21T05:46:13.133409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":129,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.209048+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:46:30.209048+0000","last_peered":"2024-09-21T05:46:30.209048+0000","last_clean":"2024-09-21T05:46:30.209048+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:46:30.209048+0000","last_undegraded":"2024-09-21T05:46:30.209048+0000","last_fullsized":"2024-09-21T05:46:30.209048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134643+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:46:13.134643+0000","last_peered":"2024-09-21T05:46:13.134643+0000","last_clean":"2024-09-21T05:46:13.134643+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:46:13.134643+0000","last_undegraded":"2024-09-21T05:46:13.134643+0000","last_fullsized":"2024-09-21T05:46:13.134643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":128,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:29.208025+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:46:29.208025+0000","last_peered":"2024-09-21T05:46:29.208025+0000","last_clean":"2024-09-21T05:46:29.208025+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:46:29.208025+0000","last_undegraded":"2024-09-21T05:46:29.208025+0000","last_fullsized":"2024-09-21T05:46:29.208025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138161+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:46:13.138161+0000","last_peered":"2024-09-21T05:46:13.138161+0000","last_clean":"2024-09-21T05:46:13.138161+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:46:13.138161+0000","last_undegraded":"2024-09-21T05:46:13.138161+0000","last_fullsized":"2024-09-21T05:46:13.138161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255492+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:45:14.255492+0000","last_peered":"2024-09-21T05:45:14.255492+0000","last_clean":"2024-09-21T05:45:14.255492+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:45:14.255492+0000","last_undegraded":"2024-09-21T05:45:14.255492+0000","last_fullsized":"2024-09-21T05:45:14.255492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.124320+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:46:13.124320+0000","last_peered":"2024-09-21T05:46:13.124320+0000","last_clean":"2024-09-21T05:46:13.124320+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:46:13.124320+0000","last_undegraded":"2024-09-21T05:46:13.124320+0000","last_fullsized":"2024-09-21T05:46:13.124320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.269448+0000","last_change":"2024-09-21T05:45:14.268855+0000","last_active":"2024-09-21T05:45:14.269448+0000","last_peered":"2024-09-21T05:45:14.269448+0000","last_clean":"2024-09-21T05:45:14.269448+0000","last_became_active":"2024-09-21T05:45:14.261558+0000","last_became_peered":"2024-09-21T05:45:14.261558+0000","last_unstale":"2024-09-21T05:45:14.269448+0000","last_undegraded":"2024-09-21T05:45:14.269448+0000","last_fullsized":"2024-09-21T05:45:14.269448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:57.978428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130304+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:46:13.130304+0000","last_peered":"2024-09-21T05:46:13.130304+0000","last_clean":"2024-09-21T05:46:13.130304+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:46:13.130304+0000","last_undegraded":"2024-09-21T05:46:13.130304+0000","last_fullsized":"2024-09-21T05:46:13.130304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"73'15","reported_seq":38,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255700+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:14.255700+0000","last_peered":"2024-09-21T05:45:14.255700+0000","last_clean":"2024-09-21T05:45:14.255700+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:14.255700+0000","last_undegraded":"2024-09-21T05:45:14.255700+0000","last_fullsized":"2024-09-21T05:45:14.255700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"73'5","reported_seq":17,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134660+0000","last_change":"2024-09-21T05:45:14.261447+0000","last_active":"2024-09-21T05:46:13.134660+0000","last_peered":"2024-09-21T05:46:13.134660+0000","last_clean":"2024-09-21T05:46:13.134660+0000","last_became_active":"2024-09-21T05:45:14.261072+0000","last_became_peered":"2024-09-21T05:45:14.261072+0000","last_unstale":"2024-09-21T05:46:13.134660+0000","last_undegraded":"2024-09-21T05:46:13.134660+0000","last_fullsized":"2024-09-21T05:46:13.134660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:46.148923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254328+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:45:14.254328+0000","last_peered":"2024-09-21T05:45:14.254328+0000","last_clean":"2024-09-21T05:45:14.254328+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:45:14.254328+0000","last_undegraded":"2024-09-21T05:45:14.254328+0000","last_fullsized":"2024-09-21T05:45:14.254328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139836+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:46:13.139836+0000","last_peered":"2024-09-21T05:46:13.139836+0000","last_clean":"2024-09-21T05:46:13.139836+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:46:13.139836+0000","last_undegraded":"2024-09-21T05:46:13.139836+0000","last_fullsized":"2024-09-21T05:46:13.139836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262195+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:45:14.262195+0000","last_peered":"2024-09-21T05:45:14.262195+0000","last_clean":"2024-09-21T05:45:14.262195+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:45:14.262195+0000","last_undegraded":"2024-09-21T05:45:14.262195+0000","last_fullsized":"2024-09-21T05:45:14.262195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.094131+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:46:13.094131+0000","last_peered":"2024-09-21T05:46:13.094131+0000","last_clean":"2024-09-21T05:46:13.094131+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:46:13.094131+0000","last_undegraded":"2024-09-21T05:46:13.094131+0000","last_fullsized":"2024-09-21T05:46:13.094131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321694+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:45:14.321694+0000","last_peered":"2024-09-21T05:45:14.321694+0000","last_clean":"2024-09-21T05:45:14.321694+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:45:14.321694+0000","last_undegraded":"2024-09-21T05:45:14.321694+0000","last_fullsized":"2024-09-21T05:45:14.321694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139062+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:46:13.139062+0000","last_peered":"2024-09-21T05:46:13.139062+0000","last_clean":"2024-09-21T05:46:13.139062+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:46:13.139062+0000","last_undegraded":"2024-09-21T05:46:13.139062+0000","last_fullsized":"2024-09-21T05:46:13.139062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256606+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:45:14.256606+0000","last_peered":"2024-09-21T05:45:14.256606+0000","last_clean":"2024-09-21T05:45:14.256606+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:45:14.256606+0000","last_undegraded":"2024-09-21T05:45:14.256606+0000","last_fullsized":"2024-09-21T05:45:14.256606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.126834+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:46:13.126834+0000","last_peered":"2024-09-21T05:46:13.126834+0000","last_clean":"2024-09-21T05:46:13.126834+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:46:13.126834+0000","last_undegraded":"2024-09-21T05:46:13.126834+0000","last_fullsized":"2024-09-21T05:46:13.126834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258860+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:45:14.258860+0000","last_peered":"2024-09-21T05:45:14.258860+0000","last_clean":"2024-09-21T05:45:14.258860+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:45:14.258860+0000","last_undegraded":"2024-09-21T05:45:14.258860+0000","last_fullsized":"2024-09-21T05:45:14.258860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135626+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:46:13.135626+0000","last_peered":"2024-09-21T05:46:13.135626+0000","last_clean":"2024-09-21T05:46:13.135626+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:46:13.135626+0000","last_undegraded":"2024-09-21T05:46:13.135626+0000","last_fullsized":"2024-09-21T05:46:13.135626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"73'15","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132962+0000","last_change":"2024-09-21T05:45:14.263694+0000","last_active":"2024-09-21T05:46:13.132962+0000","last_peered":"2024-09-21T05:46:13.132962+0000","last_clean":"2024-09-21T05:46:13.132962+0000","last_became_active":"2024-09-21T05:45:14.263394+0000","last_became_peered":"2024-09-21T05:45:14.263394+0000","last_unstale":"2024-09-21T05:46:13.132962+0000","last_undegraded":"2024-09-21T05:46:13.132962+0000","last_fullsized":"2024-09-21T05:46:13.132962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:59.214553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260360+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:45:14.260360+0000","last_peered":"2024-09-21T05:45:14.260360+0000","last_clean":"2024-09-21T05:45:14.260360+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:45:14.260360+0000","last_undegraded":"2024-09-21T05:45:14.260360+0000","last_fullsized":"2024-09-21T05:45:14.260360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134106+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:46:13.134106+0000","last_peered":"2024-09-21T05:46:13.134106+0000","last_clean":"2024-09-21T05:46:13.134106+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:46:13.134106+0000","last_undegraded":"2024-09-21T05:46:13.134106+0000","last_fullsized":"2024-09-21T05:46:13.134106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":36,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258852+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:45:14.258852+0000","last_peered":"2024-09-21T05:45:14.258852+0000","last_clean":"2024-09-21T05:45:14.258852+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:45:14.258852+0000","last_undegraded":"2024-09-21T05:45:14.258852+0000","last_fullsized":"2024-09-21T05:45:14.258852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133083+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:46:13.133083+0000","last_peered":"2024-09-21T05:46:13.133083+0000","last_clean":"2024-09-21T05:46:13.133083+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:46:13.133083+0000","last_undegraded":"2024-09-21T05:46:13.133083+0000","last_fullsized":"2024-09-21T05:46:13.133083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322148+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:45:14.322148+0000","last_peered":"2024-09-21T05:45:14.322148+0000","last_clean":"2024-09-21T05:45:14.322148+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:45:14.322148+0000","last_undegraded":"2024-09-21T05:45:14.322148+0000","last_fullsized":"2024-09-21T05:45:14.322148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130113+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:46:13.130113+0000","last_peered":"2024-09-21T05:46:13.130113+0000","last_clean":"2024-09-21T05:46:13.130113+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:46:13.130113+0000","last_undegraded":"2024-09-21T05:46:13.130113+0000","last_fullsized":"2024-09-21T05:46:13.130113+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.204331+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:46:27.204331+0000","last_peered":"2024-09-21T05:46:27.204331+0000","last_clean":"2024-09-21T05:46:27.204331+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:46:27.204331+0000","last_undegraded":"2024-09-21T05:46:27.204331+0000","last_fullsized":"2024-09-21T05:46:27.204331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77711,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1027,"num_read_kb":604,"num_write":251,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1316,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":233133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874276,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36144,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709008,"statfs":{"total":95995035648,"available":95958024192,"internally_reserved":0,"allocated":819200,"data_stored":400116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037801,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36828,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708324,"statfs":{"total":95995035648,"available":95957323776,"internally_reserved":0,"allocated":1519616,"data_stored":1060676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201325,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36156,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708996,"statfs":{"total":95995035648,"available":95958011904,"internally_reserved":0,"allocated":831488,"data_stored":398820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364850,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":819200,"data_stored":399086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561079,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1425408,"data_stored":991325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":815104,"data_stored":398537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1433600,"data_stored":990421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":823296,"data_stored":399020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":831488,"data_stored":398362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":892928,"data_stored":464894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":880640,"data_stored":464764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":835584,"data_stored":400408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":72639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":72493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:46:34.320 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-21T05:46:34.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 -- 172.21.15.28:0/3140941008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8110108c30 msgr2=0x7f811010f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 --2- 172.21.15.28:0/3140941008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8110108c30 0x7f811010f460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8108009ca0 tx=0x7f810802a8e0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 -- 172.21.15.28:0/3140941008 shutdown_connections 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 --2- 172.21.15.28:0/3140941008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8110108c30 0x7f811010f460 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 --2- 172.21.15.28:0/3140941008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101082e0 0x7f81101086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 --2- 172.21.15.28:0/3140941008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81101029b0 0x7f8110102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 -- 172.21.15.28:0/3140941008 >> 172.21.15.28:0/3140941008 conn(0x7f811007a670 msgr2=0x7f811007aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 -- 172.21.15.28:0/3140941008 shutdown_connections 2024-09-21T05:46:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.474+0000 7f8115e4d640 1 -- 172.21.15.28:0/3140941008 wait complete. 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 Processor -- start 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 -- start start 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101029b0 0x7f81101993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8110108c30 0x7f811019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81101129a0 con 0x7f81101082e0 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8110112820 con 0x7f81101029b0 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f8115e4d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8110112b20 con 0x7f8110108c30 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f810effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f810effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36896/0 (socket says 172.21.15.28:36896) 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f810effd640 1 -- 172.21.15.28:0/1535978116 learned_addr learned my addr 172.21.15.28:0/1535978116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f810f7fe640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101029b0 0x7f81101993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.475+0000 7f810ffff640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8110108c30 0x7f811019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 -- 172.21.15.28:0/1535978116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8110108c30 msgr2=0x7f811019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8110108c30 0x7f811019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 -- 172.21.15.28:0/1535978116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101029b0 msgr2=0x7f81101993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101029b0 0x7f81101993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 -- 172.21.15.28:0/1535978116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811019f2d0 con 0x7f81101082e0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810effd640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8100006fd0 tx=0x7f810000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.476+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100011830 con 0x7f81101082e0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.477+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8100016070 con 0x7f81101082e0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.477+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811019f5c0 con 0x7f81101082e0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.477+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100010e00 con 0x7f81101082e0 2024-09-21T05:46:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.477+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81101a5d50 con 0x7f81101082e0 2024-09-21T05:46:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.478+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8110102dc0 con 0x7f81101082e0 2024-09-21T05:46:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.479+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f810001b460 con 0x7f81101082e0 2024-09-21T05:46:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.480+0000 7f810cff9640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 0x7f80fc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.480+0000 7f810f7fe640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 0x7f80fc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.480+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f81000e6fa0 con 0x7f81101082e0 2024-09-21T05:46:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.481+0000 7f810f7fe640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 0x7f80fc0b8a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009bd0 tx=0x7f80f8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.484+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f81000ad730 con 0x7f81101082e0 2024-09-21T05:46:34.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:34 smithi028 ceph-mon[31227]: pgmap v311: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T05:46:34.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.671+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8110063800 con 0x7f80fc0b6560 2024-09-21T05:46:34.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.680+0000 7f810cff9640 1 -- 172.21.15.28:0/1535978116 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258879 (secure 0 0 0) 0x7f8110063800 con 0x7f80fc0b6560 2024-09-21T05:46:34.686 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:34.687 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.683+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 msgr2=0x7f80fc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.683+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 0x7f80fc0b8a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009bd0 tx=0x7f80f8005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.683+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 msgr2=0x7f8110199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.683+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8100006fd0 tx=0x7f810000b8e0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 shutdown_connections 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8110108c30 0x7f811019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80fc0b6560 0x7f80fc0b8a70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81101082e0 0x7f8110199940 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 --2- 172.21.15.28:0/1535978116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81101029b0 0x7f81101993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 >> 172.21.15.28:0/1535978116 conn(0x7f811007a670 msgr2=0x7f81101042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 shutdown_connections 2024-09-21T05:46:34.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:34.684+0000 7f8115e4d640 1 -- 172.21.15.28:0/1535978116 wait complete. 2024-09-21T05:46:34.697 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-21T05:46:32.739991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668313,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1108,"num_read_kb":673,"num_write":376,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":2953156,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394176,"kb_used_data":11648,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547648,"statfs":{"total":1151940427776,"available":1151536791552,"internally_reserved":0,"allocated":11927552,"data_stored":6766429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5751,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":77,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"3.3f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137687+0000","last_change":"2024-09-21T05:44:39.119018+0000","last_active":"2024-09-21T05:46:13.137687+0000","last_peered":"2024-09-21T05:46:13.137687+0000","last_clean":"2024-09-21T05:46:13.137687+0000","last_became_active":"2024-09-21T05:44:39.117482+0000","last_became_peered":"2024-09-21T05:44:39.117482+0000","last_unstale":"2024-09-21T05:46:13.137687+0000","last_undegraded":"2024-09-21T05:46:13.137687+0000","last_fullsized":"2024-09-21T05:46:13.137687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:54.745357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.115376+0000","last_change":"2024-09-21T05:44:39.118970+0000","last_active":"2024-09-21T05:46:13.115376+0000","last_peered":"2024-09-21T05:46:13.115376+0000","last_clean":"2024-09-21T05:46:13.115376+0000","last_became_active":"2024-09-21T05:44:39.118617+0000","last_became_peered":"2024-09-21T05:44:39.118617+0000","last_unstale":"2024-09-21T05:46:13.115376+0000","last_undegraded":"2024-09-21T05:46:13.115376+0000","last_fullsized":"2024-09-21T05:46:13.115376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:54.928856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133073+0000","last_change":"2024-09-21T05:44:39.116442+0000","last_active":"2024-09-21T05:46:13.133073+0000","last_peered":"2024-09-21T05:46:13.133073+0000","last_clean":"2024-09-21T05:46:13.133073+0000","last_became_active":"2024-09-21T05:44:39.115805+0000","last_became_peered":"2024-09-21T05:44:39.115805+0000","last_unstale":"2024-09-21T05:46:13.133073+0000","last_undegraded":"2024-09-21T05:46:13.133073+0000","last_fullsized":"2024-09-21T05:46:13.133073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:09.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134375+0000","last_change":"2024-09-21T05:44:39.116866+0000","last_active":"2024-09-21T05:46:13.134375+0000","last_peered":"2024-09-21T05:46:13.134375+0000","last_clean":"2024-09-21T05:46:13.134375+0000","last_became_active":"2024-09-21T05:44:39.116450+0000","last_became_peered":"2024-09-21T05:44:39.116450+0000","last_unstale":"2024-09-21T05:46:13.134375+0000","last_undegraded":"2024-09-21T05:46:13.134375+0000","last_fullsized":"2024-09-21T05:46:13.134375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.413159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129532+0000","last_change":"2024-09-21T05:44:39.124639+0000","last_active":"2024-09-21T05:46:13.129532+0000","last_peered":"2024-09-21T05:46:13.129532+0000","last_clean":"2024-09-21T05:46:13.129532+0000","last_became_active":"2024-09-21T05:44:39.124237+0000","last_became_peered":"2024-09-21T05:44:39.124237+0000","last_unstale":"2024-09-21T05:46:13.129532+0000","last_undegraded":"2024-09-21T05:46:13.129532+0000","last_fullsized":"2024-09-21T05:46:13.129532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:13.644477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"73'6","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128199+0000","last_change":"2024-09-21T05:45:14.258964+0000","last_active":"2024-09-21T05:46:13.128199+0000","last_peered":"2024-09-21T05:46:13.128199+0000","last_clean":"2024-09-21T05:46:13.128199+0000","last_became_active":"2024-09-21T05:45:14.257864+0000","last_became_peered":"2024-09-21T05:45:14.257864+0000","last_unstale":"2024-09-21T05:46:13.128199+0000","last_undegraded":"2024-09-21T05:46:13.128199+0000","last_fullsized":"2024-09-21T05:46:13.128199+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.693452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133810+0000","last_change":"2024-09-21T05:44:39.125383+0000","last_active":"2024-09-21T05:46:13.133810+0000","last_peered":"2024-09-21T05:46:13.133810+0000","last_clean":"2024-09-21T05:46:13.133810+0000","last_became_active":"2024-09-21T05:44:39.124983+0000","last_became_peered":"2024-09-21T05:44:39.124983+0000","last_unstale":"2024-09-21T05:46:13.133810+0000","last_undegraded":"2024-09-21T05:46:13.133810+0000","last_fullsized":"2024-09-21T05:46:13.133810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:36.010979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131962+0000","last_change":"2024-09-21T05:44:39.124753+0000","last_active":"2024-09-21T05:46:13.131962+0000","last_peered":"2024-09-21T05:46:13.131962+0000","last_clean":"2024-09-21T05:46:13.131962+0000","last_became_active":"2024-09-21T05:44:39.124382+0000","last_became_peered":"2024-09-21T05:44:39.124382+0000","last_unstale":"2024-09-21T05:46:13.131962+0000","last_undegraded":"2024-09-21T05:46:13.131962+0000","last_fullsized":"2024-09-21T05:46:13.131962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:16:23.222604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138970+0000","last_change":"2024-09-21T05:44:39.116904+0000","last_active":"2024-09-21T05:46:13.138970+0000","last_peered":"2024-09-21T05:46:13.138970+0000","last_clean":"2024-09-21T05:46:13.138970+0000","last_became_active":"2024-09-21T05:44:39.116497+0000","last_became_peered":"2024-09-21T05:44:39.116497+0000","last_unstale":"2024-09-21T05:46:13.138970+0000","last_undegraded":"2024-09-21T05:46:13.138970+0000","last_fullsized":"2024-09-21T05:46:13.138970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:14.680197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"73'16","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139172+0000","last_change":"2024-09-21T05:44:39.130265+0000","last_active":"2024-09-21T05:46:13.139172+0000","last_peered":"2024-09-21T05:46:13.139172+0000","last_clean":"2024-09-21T05:46:13.139172+0000","last_became_active":"2024-09-21T05:44:39.129871+0000","last_became_peered":"2024-09-21T05:44:39.129871+0000","last_unstale":"2024-09-21T05:46:13.139172+0000","last_undegraded":"2024-09-21T05:46:13.139172+0000","last_fullsized":"2024-09-21T05:46:13.139172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:22:17.967537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135662+0000","last_change":"2024-09-21T05:44:39.128193+0000","last_active":"2024-09-21T05:46:13.135662+0000","last_peered":"2024-09-21T05:46:13.135662+0000","last_clean":"2024-09-21T05:46:13.135662+0000","last_became_active":"2024-09-21T05:44:39.127769+0000","last_became_peered":"2024-09-21T05:44:39.127769+0000","last_unstale":"2024-09-21T05:46:13.135662+0000","last_undegraded":"2024-09-21T05:46:13.135662+0000","last_fullsized":"2024-09-21T05:46:13.135662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:50.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134240+0000","last_change":"2024-09-21T05:44:39.121591+0000","last_active":"2024-09-21T05:46:13.134240+0000","last_peered":"2024-09-21T05:46:13.134240+0000","last_clean":"2024-09-21T05:46:13.134240+0000","last_became_active":"2024-09-21T05:44:39.119348+0000","last_became_peered":"2024-09-21T05:44:39.119348+0000","last_unstale":"2024-09-21T05:46:13.134240+0000","last_undegraded":"2024-09-21T05:46:13.134240+0000","last_fullsized":"2024-09-21T05:46:13.134240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:06.847802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129338+0000","last_change":"2024-09-21T05:44:39.123558+0000","last_active":"2024-09-21T05:46:13.129338+0000","last_peered":"2024-09-21T05:46:13.129338+0000","last_clean":"2024-09-21T05:46:13.129338+0000","last_became_active":"2024-09-21T05:44:39.123044+0000","last_became_peered":"2024-09-21T05:44:39.123044+0000","last_unstale":"2024-09-21T05:46:13.129338+0000","last_undegraded":"2024-09-21T05:46:13.129338+0000","last_fullsized":"2024-09-21T05:46:13.129338+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:49:21.676368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122462+0000","last_change":"2024-09-21T05:44:39.117826+0000","last_active":"2024-09-21T05:46:13.122462+0000","last_peered":"2024-09-21T05:46:13.122462+0000","last_clean":"2024-09-21T05:46:13.122462+0000","last_became_active":"2024-09-21T05:44:39.117486+0000","last_became_peered":"2024-09-21T05:44:39.117486+0000","last_unstale":"2024-09-21T05:46:13.122462+0000","last_undegraded":"2024-09-21T05:46:13.122462+0000","last_fullsized":"2024-09-21T05:46:13.122462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:37.006829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138684+0000","last_change":"2024-09-21T05:44:39.122475+0000","last_active":"2024-09-21T05:46:13.138684+0000","last_peered":"2024-09-21T05:46:13.138684+0000","last_clean":"2024-09-21T05:46:13.138684+0000","last_became_active":"2024-09-21T05:44:39.122167+0000","last_became_peered":"2024-09-21T05:44:39.122167+0000","last_unstale":"2024-09-21T05:46:13.138684+0000","last_undegraded":"2024-09-21T05:46:13.138684+0000","last_fullsized":"2024-09-21T05:46:13.138684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:18.231671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131791+0000","last_change":"2024-09-21T05:44:39.123298+0000","last_active":"2024-09-21T05:46:13.131791+0000","last_peered":"2024-09-21T05:46:13.131791+0000","last_clean":"2024-09-21T05:46:13.131791+0000","last_became_active":"2024-09-21T05:44:39.122928+0000","last_became_peered":"2024-09-21T05:44:39.122928+0000","last_unstale":"2024-09-21T05:46:13.131791+0000","last_undegraded":"2024-09-21T05:46:13.131791+0000","last_fullsized":"2024-09-21T05:46:13.131791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:38.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.137966+0000","last_change":"2024-09-21T05:44:39.116570+0000","last_active":"2024-09-21T05:46:13.137966+0000","last_peered":"2024-09-21T05:46:13.137966+0000","last_clean":"2024-09-21T05:46:13.137966+0000","last_became_active":"2024-09-21T05:44:39.116194+0000","last_became_peered":"2024-09-21T05:44:39.116194+0000","last_unstale":"2024-09-21T05:46:13.137966+0000","last_undegraded":"2024-09-21T05:46:13.137966+0000","last_fullsized":"2024-09-21T05:46:13.137966+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:46.280127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131930+0000","last_change":"2024-09-21T05:44:39.118519+0000","last_active":"2024-09-21T05:46:13.131930+0000","last_peered":"2024-09-21T05:46:13.131930+0000","last_clean":"2024-09-21T05:46:13.131930+0000","last_became_active":"2024-09-21T05:44:39.117751+0000","last_became_peered":"2024-09-21T05:44:39.117751+0000","last_unstale":"2024-09-21T05:46:13.131930+0000","last_undegraded":"2024-09-21T05:46:13.131930+0000","last_fullsized":"2024-09-21T05:46:13.131930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:28.879539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.129054+0000","last_change":"2024-09-21T05:44:39.118887+0000","last_active":"2024-09-21T05:46:13.129054+0000","last_peered":"2024-09-21T05:46:13.129054+0000","last_clean":"2024-09-21T05:46:13.129054+0000","last_became_active":"2024-09-21T05:44:39.118416+0000","last_became_peered":"2024-09-21T05:44:39.118416+0000","last_unstale":"2024-09-21T05:46:13.129054+0000","last_undegraded":"2024-09-21T05:46:13.129054+0000","last_fullsized":"2024-09-21T05:46:13.129054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:48:03.402079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"73'8","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134973+0000","last_change":"2024-09-21T05:45:14.263089+0000","last_active":"2024-09-21T05:46:13.134973+0000","last_peered":"2024-09-21T05:46:13.134973+0000","last_clean":"2024-09-21T05:46:13.134973+0000","last_became_active":"2024-09-21T05:45:14.262626+0000","last_became_peered":"2024-09-21T05:45:14.262626+0000","last_unstale":"2024-09-21T05:46:13.134973+0000","last_undegraded":"2024-09-21T05:46:13.134973+0000","last_fullsized":"2024-09-21T05:46:13.134973+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:34.442511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130596+0000","last_change":"2024-09-21T05:44:39.123194+0000","last_active":"2024-09-21T05:46:13.130596+0000","last_peered":"2024-09-21T05:46:13.130596+0000","last_clean":"2024-09-21T05:46:13.130596+0000","last_became_active":"2024-09-21T05:44:39.122804+0000","last_became_peered":"2024-09-21T05:44:39.122804+0000","last_unstale":"2024-09-21T05:46:13.130596+0000","last_undegraded":"2024-09-21T05:46:13.130596+0000","last_fullsized":"2024-09-21T05:46:13.130596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:57.114515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132727+0000","last_change":"2024-09-21T05:44:39.124727+0000","last_active":"2024-09-21T05:46:13.132727+0000","last_peered":"2024-09-21T05:46:13.132727+0000","last_clean":"2024-09-21T05:46:13.132727+0000","last_became_active":"2024-09-21T05:44:39.124325+0000","last_became_peered":"2024-09-21T05:44:39.124325+0000","last_unstale":"2024-09-21T05:46:13.132727+0000","last_undegraded":"2024-09-21T05:46:13.132727+0000","last_fullsized":"2024-09-21T05:46:13.132727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:40:18.407104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130363+0000","last_change":"2024-09-21T05:44:39.126524+0000","last_active":"2024-09-21T05:46:13.130363+0000","last_peered":"2024-09-21T05:46:13.130363+0000","last_clean":"2024-09-21T05:46:13.130363+0000","last_became_active":"2024-09-21T05:44:39.126127+0000","last_became_peered":"2024-09-21T05:44:39.126127+0000","last_unstale":"2024-09-21T05:46:13.130363+0000","last_undegraded":"2024-09-21T05:46:13.130363+0000","last_fullsized":"2024-09-21T05:46:13.130363+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:47.291888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130982+0000","last_change":"2024-09-21T05:44:39.125605+0000","last_active":"2024-09-21T05:46:13.130982+0000","last_peered":"2024-09-21T05:46:13.130982+0000","last_clean":"2024-09-21T05:46:13.130982+0000","last_became_active":"2024-09-21T05:44:39.125227+0000","last_became_peered":"2024-09-21T05:44:39.125227+0000","last_unstale":"2024-09-21T05:46:13.130982+0000","last_undegraded":"2024-09-21T05:46:13.130982+0000","last_fullsized":"2024-09-21T05:46:13.130982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:04.270053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135513+0000","last_change":"2024-09-21T05:44:39.128580+0000","last_active":"2024-09-21T05:46:13.135513+0000","last_peered":"2024-09-21T05:46:13.135513+0000","last_clean":"2024-09-21T05:46:13.135513+0000","last_became_active":"2024-09-21T05:44:39.128046+0000","last_became_peered":"2024-09-21T05:44:39.128046+0000","last_unstale":"2024-09-21T05:46:13.135513+0000","last_undegraded":"2024-09-21T05:46:13.135513+0000","last_fullsized":"2024-09-21T05:46:13.135513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:01.496453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132358+0000","last_change":"2024-09-21T05:44:39.126598+0000","last_active":"2024-09-21T05:46:13.132358+0000","last_peered":"2024-09-21T05:46:13.132358+0000","last_clean":"2024-09-21T05:46:13.132358+0000","last_became_active":"2024-09-21T05:44:39.126224+0000","last_became_peered":"2024-09-21T05:44:39.126224+0000","last_unstale":"2024-09-21T05:46:13.132358+0000","last_undegraded":"2024-09-21T05:46:13.132358+0000","last_fullsized":"2024-09-21T05:46:13.132358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.977609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135025+0000","last_change":"2024-09-21T05:44:39.124120+0000","last_active":"2024-09-21T05:46:13.135025+0000","last_peered":"2024-09-21T05:46:13.135025+0000","last_clean":"2024-09-21T05:46:13.135025+0000","last_became_active":"2024-09-21T05:44:39.123608+0000","last_became_peered":"2024-09-21T05:44:39.123608+0000","last_unstale":"2024-09-21T05:46:13.135025+0000","last_undegraded":"2024-09-21T05:46:13.135025+0000","last_fullsized":"2024-09-21T05:46:13.135025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:30.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"73'4","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134771+0000","last_change":"2024-09-21T05:44:39.111284+0000","last_active":"2024-09-21T05:46:13.134771+0000","last_peered":"2024-09-21T05:46:13.134771+0000","last_clean":"2024-09-21T05:46:13.134771+0000","last_became_active":"2024-09-21T05:44:39.110910+0000","last_became_peered":"2024-09-21T05:44:39.110910+0000","last_unstale":"2024-09-21T05:46:13.134771+0000","last_undegraded":"2024-09-21T05:46:13.134771+0000","last_fullsized":"2024-09-21T05:46:13.134771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:14.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138706+0000","last_change":"2024-09-21T05:44:39.122577+0000","last_active":"2024-09-21T05:46:13.138706+0000","last_peered":"2024-09-21T05:46:13.138706+0000","last_clean":"2024-09-21T05:46:13.138706+0000","last_became_active":"2024-09-21T05:44:39.122241+0000","last_became_peered":"2024-09-21T05:44:39.122241+0000","last_unstale":"2024-09-21T05:46:13.138706+0000","last_undegraded":"2024-09-21T05:46:13.138706+0000","last_fullsized":"2024-09-21T05:46:13.138706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:40.327946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127051+0000","last_change":"2024-09-21T05:44:39.116761+0000","last_active":"2024-09-21T05:46:13.127051+0000","last_peered":"2024-09-21T05:46:13.127051+0000","last_clean":"2024-09-21T05:46:13.127051+0000","last_became_active":"2024-09-21T05:44:39.116285+0000","last_became_peered":"2024-09-21T05:44:39.116285+0000","last_unstale":"2024-09-21T05:46:13.127051+0000","last_undegraded":"2024-09-21T05:46:13.127051+0000","last_fullsized":"2024-09-21T05:46:13.127051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:53:35.610971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"73'12","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132243+0000","last_change":"2024-09-21T05:44:39.122819+0000","last_active":"2024-09-21T05:46:13.132243+0000","last_peered":"2024-09-21T05:46:13.132243+0000","last_clean":"2024-09-21T05:46:13.132243+0000","last_became_active":"2024-09-21T05:44:39.122381+0000","last_became_peered":"2024-09-21T05:44:39.122381+0000","last_unstale":"2024-09-21T05:46:13.132243+0000","last_undegraded":"2024-09-21T05:46:13.132243+0000","last_fullsized":"2024-09-21T05:46:13.132243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:27.267279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139372+0000","last_change":"2024-09-21T05:44:39.116532+0000","last_active":"2024-09-21T05:46:13.139372+0000","last_peered":"2024-09-21T05:46:13.139372+0000","last_clean":"2024-09-21T05:46:13.139372+0000","last_became_active":"2024-09-21T05:44:39.116022+0000","last_became_peered":"2024-09-21T05:44:39.116022+0000","last_unstale":"2024-09-21T05:46:13.139372+0000","last_undegraded":"2024-09-21T05:46:13.139372+0000","last_fullsized":"2024-09-21T05:46:13.139372+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:09.785410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.122182+0000","last_change":"2024-09-21T05:44:39.126709+0000","last_active":"2024-09-21T05:46:13.122182+0000","last_peered":"2024-09-21T05:46:13.122182+0000","last_clean":"2024-09-21T05:46:13.122182+0000","last_became_active":"2024-09-21T05:44:39.126360+0000","last_became_peered":"2024-09-21T05:44:39.126360+0000","last_unstale":"2024-09-21T05:46:13.122182+0000","last_undegraded":"2024-09-21T05:46:13.122182+0000","last_fullsized":"2024-09-21T05:46:13.122182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:36.506301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"71'5","reported_seq":144,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:31.374414+0000","last_change":"2024-09-21T05:45:14.267608+0000","last_active":"2024-09-21T05:46:31.374414+0000","last_peered":"2024-09-21T05:46:31.374414+0000","last_clean":"2024-09-21T05:46:31.374414+0000","last_became_active":"2024-09-21T05:45:14.260846+0000","last_became_peered":"2024-09-21T05:45:14.260846+0000","last_unstale":"2024-09-21T05:46:31.374414+0000","last_undegraded":"2024-09-21T05:46:31.374414+0000","last_fullsized":"2024-09-21T05:46:31.374414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:29.557266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134889+0000","last_change":"2024-09-21T05:44:39.131069+0000","last_active":"2024-09-21T05:46:13.134889+0000","last_peered":"2024-09-21T05:46:13.134889+0000","last_clean":"2024-09-21T05:46:13.134889+0000","last_became_active":"2024-09-21T05:44:39.130697+0000","last_became_peered":"2024-09-21T05:44:39.130697+0000","last_unstale":"2024-09-21T05:46:13.134889+0000","last_undegraded":"2024-09-21T05:46:13.134889+0000","last_fullsized":"2024-09-21T05:46:13.134889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.861774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"71'7","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.275469+0000","last_change":"2024-09-21T05:45:14.275384+0000","last_active":"2024-09-21T05:45:14.275469+0000","last_peered":"2024-09-21T05:45:14.275469+0000","last_clean":"2024-09-21T05:45:14.275469+0000","last_became_active":"2024-09-21T05:45:14.265194+0000","last_became_peered":"2024-09-21T05:45:14.265194+0000","last_unstale":"2024-09-21T05:45:14.275469+0000","last_undegraded":"2024-09-21T05:45:14.275469+0000","last_fullsized":"2024-09-21T05:45:14.275469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:37:24.227108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252190+0000","last_change":"2024-09-21T05:44:37.113350+0000","last_active":"2024-09-21T05:45:14.252190+0000","last_peered":"2024-09-21T05:45:14.252190+0000","last_clean":"2024-09-21T05:45:14.252190+0000","last_became_active":"2024-09-21T05:44:37.112977+0000","last_became_peered":"2024-09-21T05:44:37.112977+0000","last_unstale":"2024-09-21T05:45:14.252190+0000","last_undegraded":"2024-09-21T05:45:14.252190+0000","last_fullsized":"2024-09-21T05:45:14.252190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:52.094376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.128150+0000","last_change":"2024-09-21T05:44:39.107968+0000","last_active":"2024-09-21T05:46:13.128150+0000","last_peered":"2024-09-21T05:46:13.128150+0000","last_clean":"2024-09-21T05:46:13.128150+0000","last_became_active":"2024-09-21T05:44:39.107515+0000","last_became_peered":"2024-09-21T05:44:39.107515+0000","last_unstale":"2024-09-21T05:46:13.128150+0000","last_undegraded":"2024-09-21T05:46:13.128150+0000","last_fullsized":"2024-09-21T05:46:13.128150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:51.022087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"71'3","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.278537+0000","last_change":"2024-09-21T05:45:14.278120+0000","last_active":"2024-09-21T05:45:14.278537+0000","last_peered":"2024-09-21T05:45:14.278537+0000","last_clean":"2024-09-21T05:45:14.278537+0000","last_became_active":"2024-09-21T05:45:14.263713+0000","last_became_peered":"2024-09-21T05:45:14.263713+0000","last_unstale":"2024-09-21T05:45:14.278537+0000","last_undegraded":"2024-09-21T05:45:14.278537+0000","last_fullsized":"2024-09-21T05:45:14.278537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:16.259132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.131917+0000","last_change":"2024-09-21T05:44:39.118206+0000","last_active":"2024-09-21T05:46:13.131917+0000","last_peered":"2024-09-21T05:46:13.131917+0000","last_clean":"2024-09-21T05:46:13.131917+0000","last_became_active":"2024-09-21T05:44:39.117839+0000","last_became_peered":"2024-09-21T05:44:39.117839+0000","last_unstale":"2024-09-21T05:46:13.131917+0000","last_undegraded":"2024-09-21T05:46:13.131917+0000","last_fullsized":"2024-09-21T05:46:13.131917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:34:45.282458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"73'5","reported_seq":230,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.212715+0000","last_change":"2024-09-21T05:44:37.111449+0000","last_active":"2024-09-21T05:46:32.212715+0000","last_peered":"2024-09-21T05:46:32.212715+0000","last_clean":"2024-09-21T05:46:32.212715+0000","last_became_active":"2024-09-21T05:44:37.111099+0000","last_became_peered":"2024-09-21T05:44:37.111099+0000","last_unstale":"2024-09-21T05:46:32.212715+0000","last_undegraded":"2024-09-21T05:46:32.212715+0000","last_fullsized":"2024-09-21T05:46:32.212715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:23.496320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":202,"num_read_kb":202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"73'10","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133943+0000","last_change":"2024-09-21T05:44:39.119709+0000","last_active":"2024-09-21T05:46:13.133943+0000","last_peered":"2024-09-21T05:46:13.133943+0000","last_clean":"2024-09-21T05:46:13.133943+0000","last_became_active":"2024-09-21T05:44:39.119304+0000","last_became_peered":"2024-09-21T05:44:39.119304+0000","last_unstale":"2024-09-21T05:46:13.133943+0000","last_undegraded":"2024-09-21T05:46:13.133943+0000","last_fullsized":"2024-09-21T05:46:13.133943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:42.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"71'4","reported_seq":40,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.270877+0000","last_change":"2024-09-21T05:45:14.270775+0000","last_active":"2024-09-21T05:45:14.270877+0000","last_peered":"2024-09-21T05:45:14.270877+0000","last_clean":"2024-09-21T05:45:14.270877+0000","last_became_active":"2024-09-21T05:45:14.263657+0000","last_became_peered":"2024-09-21T05:45:14.263657+0000","last_unstale":"2024-09-21T05:45:14.270877+0000","last_undegraded":"2024-09-21T05:45:14.270877+0000","last_fullsized":"2024-09-21T05:45:14.270877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:27.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139619+0000","last_change":"2024-09-21T05:44:39.125751+0000","last_active":"2024-09-21T05:46:13.139619+0000","last_peered":"2024-09-21T05:46:13.139619+0000","last_clean":"2024-09-21T05:46:13.139619+0000","last_became_active":"2024-09-21T05:44:39.125437+0000","last_became_peered":"2024-09-21T05:44:39.125437+0000","last_unstale":"2024-09-21T05:46:13.139619+0000","last_undegraded":"2024-09-21T05:46:13.139619+0000","last_fullsized":"2024-09-21T05:46:13.139619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:02:10.900874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253308+0000","last_change":"2024-09-21T05:44:37.270754+0000","last_active":"2024-09-21T05:45:14.253308+0000","last_peered":"2024-09-21T05:45:14.253308+0000","last_clean":"2024-09-21T05:45:14.253308+0000","last_became_active":"2024-09-21T05:44:37.270235+0000","last_became_peered":"2024-09-21T05:44:37.270235+0000","last_unstale":"2024-09-21T05:45:14.253308+0000","last_undegraded":"2024-09-21T05:45:14.253308+0000","last_fullsized":"2024-09-21T05:45:14.253308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:10:12.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"73'11","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139494+0000","last_change":"2024-09-21T05:44:39.127726+0000","last_active":"2024-09-21T05:46:13.139494+0000","last_peered":"2024-09-21T05:46:13.139494+0000","last_clean":"2024-09-21T05:46:13.139494+0000","last_became_active":"2024-09-21T05:44:39.126500+0000","last_became_peered":"2024-09-21T05:44:39.126500+0000","last_unstale":"2024-09-21T05:46:13.139494+0000","last_undegraded":"2024-09-21T05:46:13.139494+0000","last_fullsized":"2024-09-21T05:46:13.139494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:44.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"73'36","reported_seq":272,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:32.375781+0000","last_change":"2024-09-21T05:44:37.114018+0000","last_active":"2024-09-21T05:46:32.375781+0000","last_peered":"2024-09-21T05:46:32.375781+0000","last_clean":"2024-09-21T05:46:32.375781+0000","last_became_active":"2024-09-21T05:44:37.113678+0000","last_became_peered":"2024-09-21T05:44:37.113678+0000","last_unstale":"2024-09-21T05:46:32.375781+0000","last_undegraded":"2024-09-21T05:46:32.375781+0000","last_fullsized":"2024-09-21T05:46:32.375781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:28.602680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70757,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":213,"num_read_kb":175,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.120849+0000","last_change":"2024-09-21T05:44:39.127616+0000","last_active":"2024-09-21T05:46:13.120849+0000","last_peered":"2024-09-21T05:46:13.120849+0000","last_clean":"2024-09-21T05:46:13.120849+0000","last_became_active":"2024-09-21T05:44:39.125840+0000","last_became_peered":"2024-09-21T05:44:39.125840+0000","last_unstale":"2024-09-21T05:46:13.120849+0000","last_undegraded":"2024-09-21T05:46:13.120849+0000","last_fullsized":"2024-09-21T05:46:13.120849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:25:27.561994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"71'9","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256407+0000","last_change":"2024-09-21T05:44:37.109509+0000","last_active":"2024-09-21T05:45:14.256407+0000","last_peered":"2024-09-21T05:45:14.256407+0000","last_clean":"2024-09-21T05:45:14.256407+0000","last_became_active":"2024-09-21T05:44:37.109011+0000","last_became_peered":"2024-09-21T05:44:37.109011+0000","last_unstale":"2024-09-21T05:45:14.256407+0000","last_undegraded":"2024-09-21T05:45:14.256407+0000","last_fullsized":"2024-09-21T05:45:14.256407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:23.163997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.127982+0000","last_change":"2024-09-21T05:44:39.120769+0000","last_active":"2024-09-21T05:46:13.127982+0000","last_peered":"2024-09-21T05:46:13.127982+0000","last_clean":"2024-09-21T05:46:13.127982+0000","last_became_active":"2024-09-21T05:44:39.120432+0000","last_became_peered":"2024-09-21T05:44:39.120432+0000","last_unstale":"2024-09-21T05:46:13.127982+0000","last_undegraded":"2024-09-21T05:46:13.127982+0000","last_fullsized":"2024-09-21T05:46:13.127982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:09.954271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253000+0000","last_change":"2024-09-21T05:44:37.110743+0000","last_active":"2024-09-21T05:45:14.253000+0000","last_peered":"2024-09-21T05:45:14.253000+0000","last_clean":"2024-09-21T05:45:14.253000+0000","last_became_active":"2024-09-21T05:44:37.110307+0000","last_became_peered":"2024-09-21T05:44:37.110307+0000","last_unstale":"2024-09-21T05:45:14.253000+0000","last_undegraded":"2024-09-21T05:45:14.253000+0000","last_fullsized":"2024-09-21T05:45:14.253000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:50.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"73'15","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.141832+0000","last_change":"2024-09-21T05:44:39.124372+0000","last_active":"2024-09-21T05:46:13.141832+0000","last_peered":"2024-09-21T05:46:13.141832+0000","last_clean":"2024-09-21T05:46:13.141832+0000","last_became_active":"2024-09-21T05:44:39.123965+0000","last_became_peered":"2024-09-21T05:44:39.123965+0000","last_unstale":"2024-09-21T05:46:13.141832+0000","last_undegraded":"2024-09-21T05:46:13.141832+0000","last_fullsized":"2024-09-21T05:46:13.141832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:58:47.301504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"71'5","reported_seq":127,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.205207+0000","last_change":"2024-09-21T05:44:37.113966+0000","last_active":"2024-09-21T05:46:27.205207+0000","last_peered":"2024-09-21T05:46:27.205207+0000","last_clean":"2024-09-21T05:46:27.205207+0000","last_became_active":"2024-09-21T05:44:37.113490+0000","last_became_peered":"2024-09-21T05:44:37.113490+0000","last_unstale":"2024-09-21T05:46:27.205207+0000","last_undegraded":"2024-09-21T05:46:27.205207+0000","last_fullsized":"2024-09-21T05:46:27.205207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:21.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133477+0000","last_change":"2024-09-21T05:44:39.120586+0000","last_active":"2024-09-21T05:46:13.133477+0000","last_peered":"2024-09-21T05:46:13.133477+0000","last_clean":"2024-09-21T05:46:13.133477+0000","last_became_active":"2024-09-21T05:44:39.120215+0000","last_became_peered":"2024-09-21T05:44:39.120215+0000","last_unstale":"2024-09-21T05:46:13.133477+0000","last_undegraded":"2024-09-21T05:46:13.133477+0000","last_fullsized":"2024-09-21T05:46:13.133477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:24.582295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"71'17","reported_seq":41,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258397+0000","last_change":"2024-09-21T05:44:37.271018+0000","last_active":"2024-09-21T05:45:14.258397+0000","last_peered":"2024-09-21T05:45:14.258397+0000","last_clean":"2024-09-21T05:45:14.258397+0000","last_became_active":"2024-09-21T05:44:37.269777+0000","last_became_peered":"2024-09-21T05:44:37.269777+0000","last_unstale":"2024-09-21T05:45:14.258397+0000","last_undegraded":"2024-09-21T05:45:14.258397+0000","last_fullsized":"2024-09-21T05:45:14.258397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:15.572137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135098+0000","last_change":"2024-09-21T05:44:39.128659+0000","last_active":"2024-09-21T05:46:13.135098+0000","last_peered":"2024-09-21T05:46:13.135098+0000","last_clean":"2024-09-21T05:46:13.135098+0000","last_became_active":"2024-09-21T05:44:39.128318+0000","last_became_peered":"2024-09-21T05:44:39.128318+0000","last_unstale":"2024-09-21T05:46:13.135098+0000","last_undegraded":"2024-09-21T05:46:13.135098+0000","last_fullsized":"2024-09-21T05:46:13.135098+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:04.390483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"71'2","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.259227+0000","last_change":"2024-09-21T05:44:37.113604+0000","last_active":"2024-09-21T05:45:14.259227+0000","last_peered":"2024-09-21T05:45:14.259227+0000","last_clean":"2024-09-21T05:45:14.259227+0000","last_became_active":"2024-09-21T05:44:37.113161+0000","last_became_peered":"2024-09-21T05:44:37.113161+0000","last_unstale":"2024-09-21T05:45:14.259227+0000","last_undegraded":"2024-09-21T05:45:14.259227+0000","last_fullsized":"2024-09-21T05:45:14.259227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:22.687133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"73'5","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134830+0000","last_change":"2024-09-21T05:45:14.263088+0000","last_active":"2024-09-21T05:46:13.134830+0000","last_peered":"2024-09-21T05:46:13.134830+0000","last_clean":"2024-09-21T05:46:13.134830+0000","last_became_active":"2024-09-21T05:45:14.262643+0000","last_became_peered":"2024-09-21T05:45:14.262643+0000","last_unstale":"2024-09-21T05:46:13.134830+0000","last_undegraded":"2024-09-21T05:46:13.134830+0000","last_fullsized":"2024-09-21T05:46:13.134830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:05.991971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"71'5","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.252502+0000","last_change":"2024-09-21T05:44:37.111270+0000","last_active":"2024-09-21T05:45:14.252502+0000","last_peered":"2024-09-21T05:45:14.252502+0000","last_clean":"2024-09-21T05:45:14.252502+0000","last_became_active":"2024-09-21T05:44:37.110831+0000","last_became_peered":"2024-09-21T05:44:37.110831+0000","last_unstale":"2024-09-21T05:45:14.252502+0000","last_undegraded":"2024-09-21T05:45:14.252502+0000","last_fullsized":"2024-09-21T05:45:14.252502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:09.328459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130462+0000","last_change":"2024-09-21T05:44:39.127324+0000","last_active":"2024-09-21T05:46:13.130462+0000","last_peered":"2024-09-21T05:46:13.130462+0000","last_clean":"2024-09-21T05:46:13.130462+0000","last_became_active":"2024-09-21T05:44:39.125854+0000","last_became_peered":"2024-09-21T05:44:39.125854+0000","last_unstale":"2024-09-21T05:46:13.130462+0000","last_undegraded":"2024-09-21T05:46:13.130462+0000","last_fullsized":"2024-09-21T05:46:13.130462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:38.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"71'3","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253487+0000","last_change":"2024-09-21T05:44:37.110196+0000","last_active":"2024-09-21T05:45:14.253487+0000","last_peered":"2024-09-21T05:45:14.253487+0000","last_clean":"2024-09-21T05:45:14.253487+0000","last_became_active":"2024-09-21T05:44:37.109438+0000","last_became_peered":"2024-09-21T05:44:37.109438+0000","last_unstale":"2024-09-21T05:45:14.253487+0000","last_undegraded":"2024-09-21T05:45:14.253487+0000","last_fullsized":"2024-09-21T05:45:14.253487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133489+0000","last_change":"2024-09-21T05:44:39.126861+0000","last_active":"2024-09-21T05:46:13.133489+0000","last_peered":"2024-09-21T05:46:13.133489+0000","last_clean":"2024-09-21T05:46:13.133489+0000","last_became_active":"2024-09-21T05:44:39.126435+0000","last_became_peered":"2024-09-21T05:44:39.126435+0000","last_unstale":"2024-09-21T05:46:13.133489+0000","last_undegraded":"2024-09-21T05:46:13.133489+0000","last_fullsized":"2024-09-21T05:46:13.133489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:00.161080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"73'7","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.210435+0000","last_change":"2024-09-21T05:44:37.530623+0000","last_active":"2024-09-21T05:46:30.210435+0000","last_peered":"2024-09-21T05:46:30.210435+0000","last_clean":"2024-09-21T05:46:30.210435+0000","last_became_active":"2024-09-21T05:44:37.530143+0000","last_became_peered":"2024-09-21T05:44:37.530143+0000","last_unstale":"2024-09-21T05:46:30.210435+0000","last_undegraded":"2024-09-21T05:46:30.210435+0000","last_fullsized":"2024-09-21T05:46:30.210435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:57.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135958+0000","last_change":"2024-09-21T05:44:39.122628+0000","last_active":"2024-09-21T05:46:13.135958+0000","last_peered":"2024-09-21T05:46:13.135958+0000","last_clean":"2024-09-21T05:46:13.135958+0000","last_became_active":"2024-09-21T05:44:39.122236+0000","last_became_peered":"2024-09-21T05:44:39.122236+0000","last_unstale":"2024-09-21T05:46:13.135958+0000","last_undegraded":"2024-09-21T05:46:13.135958+0000","last_fullsized":"2024-09-21T05:46:13.135958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:16.875713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.257073+0000","last_change":"2024-09-21T05:43:28.781052+0000","last_active":"2024-09-21T05:45:14.257073+0000","last_peered":"2024-09-21T05:45:14.257073+0000","last_clean":"2024-09-21T05:45:14.257073+0000","last_became_active":"2024-09-21T05:43:28.774896+0000","last_became_peered":"2024-09-21T05:43:28.774896+0000","last_unstale":"2024-09-21T05:45:14.257073+0000","last_undegraded":"2024-09-21T05:45:14.257073+0000","last_fullsized":"2024-09-21T05:45:14.257073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:40:11.288414+0000","last_clean_scrub_stamp":"2024-09-21T05:40:11.288414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:53.493835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"71'6","reported_seq":32,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.253471+0000","last_change":"2024-09-21T05:44:37.112212+0000","last_active":"2024-09-21T05:45:14.253471+0000","last_peered":"2024-09-21T05:45:14.253471+0000","last_clean":"2024-09-21T05:45:14.253471+0000","last_became_active":"2024-09-21T05:44:37.111327+0000","last_became_peered":"2024-09-21T05:44:37.111327+0000","last_unstale":"2024-09-21T05:45:14.253471+0000","last_undegraded":"2024-09-21T05:45:14.253471+0000","last_fullsized":"2024-09-21T05:45:14.253471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:21.691291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130028+0000","last_change":"2024-09-21T05:44:39.130149+0000","last_active":"2024-09-21T05:46:13.130028+0000","last_peered":"2024-09-21T05:46:13.130028+0000","last_clean":"2024-09-21T05:46:13.130028+0000","last_became_active":"2024-09-21T05:44:39.129711+0000","last_became_peered":"2024-09-21T05:44:39.129711+0000","last_unstale":"2024-09-21T05:46:13.130028+0000","last_undegraded":"2024-09-21T05:46:13.130028+0000","last_fullsized":"2024-09-21T05:46:13.130028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:29.119222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"71'4","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256020+0000","last_change":"2024-09-21T05:44:37.108461+0000","last_active":"2024-09-21T05:45:14.256020+0000","last_peered":"2024-09-21T05:45:14.256020+0000","last_clean":"2024-09-21T05:45:14.256020+0000","last_became_active":"2024-09-21T05:44:37.108042+0000","last_became_peered":"2024-09-21T05:44:37.108042+0000","last_unstale":"2024-09-21T05:45:14.256020+0000","last_undegraded":"2024-09-21T05:45:14.256020+0000","last_fullsized":"2024-09-21T05:45:14.256020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:37.650896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133409+0000","last_change":"2024-09-21T05:44:39.113746+0000","last_active":"2024-09-21T05:46:13.133409+0000","last_peered":"2024-09-21T05:46:13.133409+0000","last_clean":"2024-09-21T05:46:13.133409+0000","last_became_active":"2024-09-21T05:44:39.113376+0000","last_became_peered":"2024-09-21T05:44:39.113376+0000","last_unstale":"2024-09-21T05:46:13.133409+0000","last_undegraded":"2024-09-21T05:46:13.133409+0000","last_fullsized":"2024-09-21T05:46:13.133409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:19:02.949447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"71'5","reported_seq":129,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:30.209048+0000","last_change":"2024-09-21T05:44:37.270214+0000","last_active":"2024-09-21T05:46:30.209048+0000","last_peered":"2024-09-21T05:46:30.209048+0000","last_clean":"2024-09-21T05:46:30.209048+0000","last_became_active":"2024-09-21T05:44:37.269784+0000","last_became_peered":"2024-09-21T05:44:37.269784+0000","last_unstale":"2024-09-21T05:46:30.209048+0000","last_undegraded":"2024-09-21T05:46:30.209048+0000","last_fullsized":"2024-09-21T05:46:30.209048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:59.496506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134643+0000","last_change":"2024-09-21T05:44:39.122587+0000","last_active":"2024-09-21T05:46:13.134643+0000","last_peered":"2024-09-21T05:46:13.134643+0000","last_clean":"2024-09-21T05:46:13.134643+0000","last_became_active":"2024-09-21T05:44:39.122218+0000","last_became_peered":"2024-09-21T05:44:39.122218+0000","last_unstale":"2024-09-21T05:46:13.134643+0000","last_undegraded":"2024-09-21T05:46:13.134643+0000","last_fullsized":"2024-09-21T05:46:13.134643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:51.903353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"71'5","reported_seq":128,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:29.208025+0000","last_change":"2024-09-21T05:44:37.108931+0000","last_active":"2024-09-21T05:46:29.208025+0000","last_peered":"2024-09-21T05:46:29.208025+0000","last_clean":"2024-09-21T05:46:29.208025+0000","last_became_active":"2024-09-21T05:44:37.108368+0000","last_became_peered":"2024-09-21T05:44:37.108368+0000","last_unstale":"2024-09-21T05:46:29.208025+0000","last_undegraded":"2024-09-21T05:46:29.208025+0000","last_fullsized":"2024-09-21T05:46:29.208025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:21:27.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.138161+0000","last_change":"2024-09-21T05:44:39.124516+0000","last_active":"2024-09-21T05:46:13.138161+0000","last_peered":"2024-09-21T05:46:13.138161+0000","last_clean":"2024-09-21T05:46:13.138161+0000","last_became_active":"2024-09-21T05:44:39.124078+0000","last_became_peered":"2024-09-21T05:44:39.124078+0000","last_unstale":"2024-09-21T05:46:13.138161+0000","last_undegraded":"2024-09-21T05:46:13.138161+0000","last_fullsized":"2024-09-21T05:46:13.138161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:43:43.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"71'1","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255492+0000","last_change":"2024-09-21T05:44:37.271211+0000","last_active":"2024-09-21T05:45:14.255492+0000","last_peered":"2024-09-21T05:45:14.255492+0000","last_clean":"2024-09-21T05:45:14.255492+0000","last_became_active":"2024-09-21T05:44:37.270693+0000","last_became_peered":"2024-09-21T05:44:37.270693+0000","last_unstale":"2024-09-21T05:45:14.255492+0000","last_undegraded":"2024-09-21T05:45:14.255492+0000","last_fullsized":"2024-09-21T05:45:14.255492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:25.421007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.124320+0000","last_change":"2024-09-21T05:44:39.124525+0000","last_active":"2024-09-21T05:46:13.124320+0000","last_peered":"2024-09-21T05:46:13.124320+0000","last_clean":"2024-09-21T05:46:13.124320+0000","last_became_active":"2024-09-21T05:44:39.123879+0000","last_became_peered":"2024-09-21T05:44:39.123879+0000","last_unstale":"2024-09-21T05:46:13.124320+0000","last_undegraded":"2024-09-21T05:46:13.124320+0000","last_fullsized":"2024-09-21T05:46:13.124320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:02.992750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"71'4","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.269448+0000","last_change":"2024-09-21T05:45:14.268855+0000","last_active":"2024-09-21T05:45:14.269448+0000","last_peered":"2024-09-21T05:45:14.269448+0000","last_clean":"2024-09-21T05:45:14.269448+0000","last_became_active":"2024-09-21T05:45:14.261558+0000","last_became_peered":"2024-09-21T05:45:14.261558+0000","last_unstale":"2024-09-21T05:45:14.269448+0000","last_undegraded":"2024-09-21T05:45:14.269448+0000","last_fullsized":"2024-09-21T05:45:14.269448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:57.978428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130304+0000","last_change":"2024-09-21T05:44:39.114232+0000","last_active":"2024-09-21T05:46:13.130304+0000","last_peered":"2024-09-21T05:46:13.130304+0000","last_clean":"2024-09-21T05:46:13.130304+0000","last_became_active":"2024-09-21T05:44:39.113049+0000","last_became_peered":"2024-09-21T05:44:39.113049+0000","last_unstale":"2024-09-21T05:46:13.130304+0000","last_undegraded":"2024-09-21T05:46:13.130304+0000","last_fullsized":"2024-09-21T05:46:13.130304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:46:17.399496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"73'15","reported_seq":38,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.255700+0000","last_change":"2024-09-21T05:44:37.103183+0000","last_active":"2024-09-21T05:45:14.255700+0000","last_peered":"2024-09-21T05:45:14.255700+0000","last_clean":"2024-09-21T05:45:14.255700+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:45:14.255700+0000","last_undegraded":"2024-09-21T05:45:14.255700+0000","last_fullsized":"2024-09-21T05:45:14.255700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:45:35.599904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"73'5","reported_seq":17,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134660+0000","last_change":"2024-09-21T05:45:14.261447+0000","last_active":"2024-09-21T05:46:13.134660+0000","last_peered":"2024-09-21T05:46:13.134660+0000","last_clean":"2024-09-21T05:46:13.134660+0000","last_became_active":"2024-09-21T05:45:14.261072+0000","last_became_peered":"2024-09-21T05:45:14.261072+0000","last_unstale":"2024-09-21T05:46:13.134660+0000","last_undegraded":"2024-09-21T05:46:13.134660+0000","last_fullsized":"2024-09-21T05:46:13.134660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:46.148923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"71'7","reported_seq":31,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.254328+0000","last_change":"2024-09-21T05:44:37.530528+0000","last_active":"2024-09-21T05:45:14.254328+0000","last_peered":"2024-09-21T05:45:14.254328+0000","last_clean":"2024-09-21T05:45:14.254328+0000","last_became_active":"2024-09-21T05:44:37.530236+0000","last_became_peered":"2024-09-21T05:44:37.530236+0000","last_unstale":"2024-09-21T05:45:14.254328+0000","last_undegraded":"2024-09-21T05:45:14.254328+0000","last_fullsized":"2024-09-21T05:45:14.254328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:06.660362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"73'7","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139836+0000","last_change":"2024-09-21T05:44:39.121350+0000","last_active":"2024-09-21T05:46:13.139836+0000","last_peered":"2024-09-21T05:46:13.139836+0000","last_clean":"2024-09-21T05:46:13.139836+0000","last_became_active":"2024-09-21T05:44:39.120702+0000","last_became_peered":"2024-09-21T05:44:39.120702+0000","last_unstale":"2024-09-21T05:46:13.139836+0000","last_undegraded":"2024-09-21T05:46:13.139836+0000","last_fullsized":"2024-09-21T05:46:13.139836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:06:43.306506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.262195+0000","last_change":"2024-09-21T05:44:37.270486+0000","last_active":"2024-09-21T05:45:14.262195+0000","last_peered":"2024-09-21T05:45:14.262195+0000","last_clean":"2024-09-21T05:45:14.262195+0000","last_became_active":"2024-09-21T05:44:37.270042+0000","last_became_peered":"2024-09-21T05:44:37.270042+0000","last_unstale":"2024-09-21T05:45:14.262195+0000","last_undegraded":"2024-09-21T05:45:14.262195+0000","last_fullsized":"2024-09-21T05:45:14.262195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:45.382770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"73'2","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.094131+0000","last_change":"2024-09-21T05:44:39.125460+0000","last_active":"2024-09-21T05:46:13.094131+0000","last_peered":"2024-09-21T05:46:13.094131+0000","last_clean":"2024-09-21T05:46:13.094131+0000","last_became_active":"2024-09-21T05:44:39.125108+0000","last_became_peered":"2024-09-21T05:44:39.125108+0000","last_unstale":"2024-09-21T05:46:13.094131+0000","last_undegraded":"2024-09-21T05:46:13.094131+0000","last_fullsized":"2024-09-21T05:46:13.094131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:14.823301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"71'7","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.321694+0000","last_change":"2024-09-21T05:44:37.108922+0000","last_active":"2024-09-21T05:45:14.321694+0000","last_peered":"2024-09-21T05:45:14.321694+0000","last_clean":"2024-09-21T05:45:14.321694+0000","last_became_active":"2024-09-21T05:44:37.108464+0000","last_became_peered":"2024-09-21T05:44:37.108464+0000","last_unstale":"2024-09-21T05:45:14.321694+0000","last_undegraded":"2024-09-21T05:45:14.321694+0000","last_fullsized":"2024-09-21T05:45:14.321694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:43.460020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.139062+0000","last_change":"2024-09-21T05:44:39.112994+0000","last_active":"2024-09-21T05:46:13.139062+0000","last_peered":"2024-09-21T05:46:13.139062+0000","last_clean":"2024-09-21T05:46:13.139062+0000","last_became_active":"2024-09-21T05:44:39.112486+0000","last_became_peered":"2024-09-21T05:44:39.112486+0000","last_unstale":"2024-09-21T05:46:13.139062+0000","last_undegraded":"2024-09-21T05:46:13.139062+0000","last_fullsized":"2024-09-21T05:46:13.139062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:06.324173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"71'3","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.256606+0000","last_change":"2024-09-21T05:44:37.108510+0000","last_active":"2024-09-21T05:45:14.256606+0000","last_peered":"2024-09-21T05:45:14.256606+0000","last_clean":"2024-09-21T05:45:14.256606+0000","last_became_active":"2024-09-21T05:44:37.108144+0000","last_became_peered":"2024-09-21T05:44:37.108144+0000","last_unstale":"2024-09-21T05:45:14.256606+0000","last_undegraded":"2024-09-21T05:45:14.256606+0000","last_fullsized":"2024-09-21T05:45:14.256606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:10.985952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"73'8","reported_seq":26,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.126834+0000","last_change":"2024-09-21T05:44:39.121043+0000","last_active":"2024-09-21T05:46:13.126834+0000","last_peered":"2024-09-21T05:46:13.126834+0000","last_clean":"2024-09-21T05:46:13.126834+0000","last_became_active":"2024-09-21T05:44:39.120676+0000","last_became_peered":"2024-09-21T05:44:39.120676+0000","last_unstale":"2024-09-21T05:46:13.126834+0000","last_undegraded":"2024-09-21T05:46:13.126834+0000","last_fullsized":"2024-09-21T05:46:13.126834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:49:52.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"71'4","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258860+0000","last_change":"2024-09-21T05:44:37.111653+0000","last_active":"2024-09-21T05:45:14.258860+0000","last_peered":"2024-09-21T05:45:14.258860+0000","last_clean":"2024-09-21T05:45:14.258860+0000","last_became_active":"2024-09-21T05:44:37.111083+0000","last_became_peered":"2024-09-21T05:44:37.111083+0000","last_unstale":"2024-09-21T05:45:14.258860+0000","last_undegraded":"2024-09-21T05:45:14.258860+0000","last_fullsized":"2024-09-21T05:45:14.258860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:33:30.652449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.135626+0000","last_change":"2024-09-21T05:44:39.118125+0000","last_active":"2024-09-21T05:46:13.135626+0000","last_peered":"2024-09-21T05:46:13.135626+0000","last_clean":"2024-09-21T05:46:13.135626+0000","last_became_active":"2024-09-21T05:44:39.117692+0000","last_became_peered":"2024-09-21T05:44:39.117692+0000","last_unstale":"2024-09-21T05:46:13.135626+0000","last_undegraded":"2024-09-21T05:46:13.135626+0000","last_fullsized":"2024-09-21T05:46:13.135626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:37.475654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"73'15","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.132962+0000","last_change":"2024-09-21T05:45:14.263694+0000","last_active":"2024-09-21T05:46:13.132962+0000","last_peered":"2024-09-21T05:46:13.132962+0000","last_clean":"2024-09-21T05:46:13.132962+0000","last_became_active":"2024-09-21T05:45:14.263394+0000","last_became_peered":"2024-09-21T05:45:14.263394+0000","last_unstale":"2024-09-21T05:46:13.132962+0000","last_undegraded":"2024-09-21T05:46:13.132962+0000","last_fullsized":"2024-09-21T05:46:13.132962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:59.214553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"71'5","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.260360+0000","last_change":"2024-09-21T05:44:37.108312+0000","last_active":"2024-09-21T05:45:14.260360+0000","last_peered":"2024-09-21T05:45:14.260360+0000","last_clean":"2024-09-21T05:45:14.260360+0000","last_became_active":"2024-09-21T05:44:37.107358+0000","last_became_peered":"2024-09-21T05:44:37.107358+0000","last_unstale":"2024-09-21T05:45:14.260360+0000","last_undegraded":"2024-09-21T05:45:14.260360+0000","last_fullsized":"2024-09-21T05:45:14.260360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:34.877011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"73'9","reported_seq":27,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.134106+0000","last_change":"2024-09-21T05:44:39.121533+0000","last_active":"2024-09-21T05:46:13.134106+0000","last_peered":"2024-09-21T05:46:13.134106+0000","last_clean":"2024-09-21T05:46:13.134106+0000","last_became_active":"2024-09-21T05:44:39.120782+0000","last_became_peered":"2024-09-21T05:44:39.120782+0000","last_unstale":"2024-09-21T05:46:13.134106+0000","last_undegraded":"2024-09-21T05:46:13.134106+0000","last_fullsized":"2024-09-21T05:46:13.134106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:28.569178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"71'12","reported_seq":36,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.258852+0000","last_change":"2024-09-21T05:44:37.107940+0000","last_active":"2024-09-21T05:45:14.258852+0000","last_peered":"2024-09-21T05:45:14.258852+0000","last_clean":"2024-09-21T05:45:14.258852+0000","last_became_active":"2024-09-21T05:44:37.107520+0000","last_became_peered":"2024-09-21T05:44:37.107520+0000","last_unstale":"2024-09-21T05:45:14.258852+0000","last_undegraded":"2024-09-21T05:45:14.258852+0000","last_fullsized":"2024-09-21T05:45:14.258852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:23.415928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"73'6","reported_seq":24,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.133083+0000","last_change":"2024-09-21T05:44:39.116434+0000","last_active":"2024-09-21T05:46:13.133083+0000","last_peered":"2024-09-21T05:46:13.133083+0000","last_clean":"2024-09-21T05:46:13.133083+0000","last_became_active":"2024-09-21T05:44:39.115922+0000","last_became_peered":"2024-09-21T05:44:39.115922+0000","last_unstale":"2024-09-21T05:46:13.133083+0000","last_undegraded":"2024-09-21T05:46:13.133083+0000","last_fullsized":"2024-09-21T05:46:13.133083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:33.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"71'2","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:45:14.322148+0000","last_change":"2024-09-21T05:44:37.270285+0000","last_active":"2024-09-21T05:45:14.322148+0000","last_peered":"2024-09-21T05:45:14.322148+0000","last_clean":"2024-09-21T05:45:14.322148+0000","last_became_active":"2024-09-21T05:44:37.269729+0000","last_became_peered":"2024-09-21T05:44:37.269729+0000","last_unstale":"2024-09-21T05:45:14.322148+0000","last_undegraded":"2024-09-21T05:45:14.322148+0000","last_fullsized":"2024-09-21T05:45:14.322148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:27:58.013726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"73'5","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:13.130113+0000","last_change":"2024-09-21T05:44:39.129201+0000","last_active":"2024-09-21T05:46:13.130113+0000","last_peered":"2024-09-21T05:46:13.130113+0000","last_clean":"2024-09-21T05:46:13.130113+0000","last_became_active":"2024-09-21T05:44:39.128800+0000","last_became_peered":"2024-09-21T05:44:39.128800+0000","last_unstale":"2024-09-21T05:46:13.130113+0000","last_undegraded":"2024-09-21T05:46:13.130113+0000","last_fullsized":"2024-09-21T05:46:13.130113+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:38.094816+0000","last_clean_scrub_stamp":"2024-09-21T05:44:38.094816+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:56:43.071332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"71'13","reported_seq":134,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T05:46:27.204331+0000","last_change":"2024-09-21T05:44:37.103512+0000","last_active":"2024-09-21T05:46:27.204331+0000","last_peered":"2024-09-21T05:46:27.204331+0000","last_clean":"2024-09-21T05:46:27.204331+0000","last_became_active":"2024-09-21T05:44:37.102797+0000","last_became_peered":"2024-09-21T05:44:37.102797+0000","last_unstale":"2024-09-21T05:46:27.204331+0000","last_undegraded":"2024-09-21T05:46:27.204331+0000","last_fullsized":"2024-09-21T05:46:27.204331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T05:44:36.080495+0000","last_clean_scrub_stamp":"2024-09-21T05:44:36.080495+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:54:46.406842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77711,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1027,"num_read_kb":604,"num_write":251,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1316,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":233133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":65,"seq":279172874276,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36144,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709008,"statfs":{"total":95995035648,"available":95958024192,"internally_reserved":0,"allocated":819200,"data_stored":400116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":60,"seq":257698037801,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36828,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708324,"statfs":{"total":95995035648,"available":95957323776,"internally_reserved":0,"allocated":1519616,"data_stored":1060676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":55,"seq":236223201325,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36156,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708996,"statfs":{"total":95995035648,"available":95958011904,"internally_reserved":0,"allocated":831488,"data_stored":398820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":50,"seq":214748364850,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":819200,"data_stored":399086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":44,"seq":188978561079,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1425408,"data_stored":991325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":39,"seq":167503724604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":815104,"data_stored":398537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":34,"seq":146028888129,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1433600,"data_stored":990421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":823296,"data_stored":399020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":831488,"data_stored":398362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":892928,"data_stored":464894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":880640,"data_stored":464764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":835584,"data_stored":400408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":72639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":72493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T05:46:34.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:46:34.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:34 smithi181 ceph-mon[34819]: pgmap v311: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T05:46:34.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:34 smithi129 ceph-mon[34885]: pgmap v311: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T05:46:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 -- 172.21.15.28:0/2303572344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 msgr2=0x7fa3fc110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 --2- 172.21.15.28:0/2303572344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc110330 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009ca0 tx=0x7fa3f402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 -- 172.21.15.28:0/2303572344 shutdown_connections 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 --2- 172.21.15.28:0/2303572344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc110330 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 --2- 172.21.15.28:0/2303572344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc105380 0x7fa3fc10d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 --2- 172.21.15.28:0/2303572344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc104a30 0x7fa3fc104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 -- 172.21.15.28:0/2303572344 >> 172.21.15.28:0/2303572344 conn(0x7fa3fc079760 msgr2=0x7fa3fc079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 -- 172.21.15.28:0/2303572344 shutdown_connections 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.475+0000 7fa400eba640 1 -- 172.21.15.28:0/2303572344 wait complete. 2024-09-21T05:46:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 Processor -- start 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 -- start start 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc104a30 0x7fa3fc19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc105380 0x7fa3fc19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc113ac0 con 0x7fa3fc10de60 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc113940 con 0x7fa3fc105380 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.476+0000 7fa400eba640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc113c40 con 0x7fa3fc104a30 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47826/0 (socket says 172.21.15.28:47826) 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 -- 172.21.15.28:0/4125041112 learned_addr learned my addr 172.21.15.28:0/4125041112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fa59c640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc104a30 0x7fa3fc19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3f9d9b640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc105380 0x7fa3fc19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 -- 172.21.15.28:0/4125041112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc104a30 msgr2=0x7fa3fc19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc104a30 0x7fa3fc19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 -- 172.21.15.28:0/4125041112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc105380 msgr2=0x7fa3fc19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc105380 0x7fa3fc19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 -- 172.21.15.28:0/4125041112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc1a03e0 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.477+0000 7fa3fad9d640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4005cc0 tx=0x7fa3f4031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.478+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003b70 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.478+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f40318f0 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.478+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f402f950 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.478+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc1a0670 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.478+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc1a6e30 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.479+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3fc06c1f0 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.481+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7fa3f4004b40 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.482+0000 7fa3eb7fe640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 0x7fa3e40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.482+0000 7fa3fa59c640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 0x7fa3e40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.483+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa3f4108680 con 0x7fa3fc10de60 2024-09-21T05:46:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.483+0000 7fa3fa59c640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 0x7fa3e40b8af0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06c8f0 tx=0x7fa3ec005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.486+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa3f40cee10 con 0x7fa3fc10de60 2024-09-21T05:46:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.710+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3fc104e40 con 0x7fa3fc10de60 2024-09-21T05:46:35.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:35 smithi028 ceph-mon[31227]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:35.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:35 smithi028 ceph-mon[31227]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.711+0000 7fa3eb7fe640 1 -- 172.21.15.28:0/4125041112 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7fa3f40d31f0 con 0x7fa3fc10de60 2024-09-21T05:46:35.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:35.713 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:46:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 msgr2=0x7fa3e40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 0x7fa3e40b8af0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06c8f0 tx=0x7fa3ec005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 msgr2=0x7fa3fc19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4005cc0 tx=0x7fa3f4031700 comp rx=0 tx=0).stop 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 shutdown_connections 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa3e40b65e0 0x7fa3e40b8af0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc10de60 0x7fa3fc19fcb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3fc105380 0x7fa3fc19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 --2- 172.21.15.28:0/4125041112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3fc104a30 0x7fa3fc19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 >> 172.21.15.28:0/4125041112 conn(0x7fa3fc079760 msgr2=0x7fa3fc1007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 shutdown_connections 2024-09-21T05:46:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:35.713+0000 7fa400eba640 1 -- 172.21.15.28:0/4125041112 wait complete. 2024-09-21T05:46:35.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:35 smithi181 ceph-mon[34819]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:35 smithi181 ceph-mon[34819]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:35.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:35 smithi129 ceph-mon[34885]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:35.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:35 smithi129 ceph-mon[34885]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:36.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:46:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:36 smithi181 ceph-mon[34819]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:36 smithi181 ceph-mon[34819]: pgmap v312: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:46:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:36 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4125041112' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:36 smithi028 ceph-mon[31227]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:36 smithi028 ceph-mon[31227]: pgmap v312: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:46:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4125041112' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:36 smithi129 ceph-mon[34885]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T05:46:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:36 smithi129 ceph-mon[34885]: pgmap v312: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:46:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:36 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4125041112' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 -- 172.21.15.28:0/1084298325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690108d10 msgr2=0x7f969010f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 --2- 172.21.15.28:0/1084298325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690108d10 0x7f969010f540 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9688009ca0 tx=0x7f968802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 -- 172.21.15.28:0/1084298325 shutdown_connections 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 --2- 172.21.15.28:0/1084298325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690108d10 0x7f969010f540 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 --2- 172.21.15.28:0/1084298325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96901083c0 0x7f96901087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 --2- 172.21.15.28:0/1084298325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 0x7f9690102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 -- 172.21.15.28:0/1084298325 >> 172.21.15.28:0/1084298325 conn(0x7f9690079740 msgr2=0x7f9690079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 -- 172.21.15.28:0/1084298325 shutdown_connections 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.172+0000 7f9696944640 1 -- 172.21.15.28:0/1084298325 wait complete. 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 Processor -- start 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 -- start start 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 0x7f9690199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9690108d10 0x7f969019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112980 con 0x7f96901083c0 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112800 con 0x7f9690102a80 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9696944640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112b00 con 0x7f9690108d10 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9687fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9687fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47852/0 (socket says 172.21.15.28:47852) 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9687fff640 1 -- 172.21.15.28:0/846732754 learned_addr learned my addr 172.21.15.28:0/846732754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f9694ee1640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9690108d10 0x7f969019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.173+0000 7f968ffff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 0x7f9690199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 -- 172.21.15.28:0/846732754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9690108d10 msgr2=0x7f969019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9690108d10 0x7f969019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 -- 172.21.15.28:0/846732754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 msgr2=0x7f9690199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 0x7f9690199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 -- 172.21.15.28:0/846732754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969019f2c0 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9687fff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f967800d740 tx=0x7f967800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678017070 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f967800de50 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969019f5b0 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678004ce0 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.174+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901a5d10 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.175+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9690103d00 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.177+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f9678004e80 con 0x7f96901083c0 2024-09-21T05:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.178+0000 7f968dffb640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 0x7f967c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.178+0000 7f968ffff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 0x7f967c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.178+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f96780e70d0 con 0x7f96901083c0 2024-09-21T05:46:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.178+0000 7f968ffff640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 0x7f967c0b8a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f968000bbe0 tx=0x7f968000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.181+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f96780b1ce0 con 0x7f96901083c0 2024-09-21T05:46:37.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.370+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f969010ab30 con 0x7f96901083c0 2024-09-21T05:46:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.371+0000 7f968dffb640 1 -- 172.21.15.28:0/846732754 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+23122 (secure 0 0 0) 0x7f96780b6c20 con 0x7f96901083c0 2024-09-21T05:46:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:46:37.374 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:45:14.248651+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:46:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 msgr2=0x7f967c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 0x7f967c0b8a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f968000bbe0 tx=0x7f968000b390 comp rx=0 tx=0).stop 2024-09-21T05:46:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 msgr2=0x7f96901998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f967800d740 tx=0x7f967800dc00 comp rx=0 tx=0).stop 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 shutdown_connections 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9690108d10 0x7f969019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f967c0b6560 0x7f967c0b8a70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96901083c0 0x7f96901998f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 --2- 172.21.15.28:0/846732754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9690102a80 0x7f9690199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 >> 172.21.15.28:0/846732754 conn(0x7f9690079740 msgr2=0x7f96901043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 shutdown_connections 2024-09-21T05:46:37.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:37.374+0000 7f9696944640 1 -- 172.21.15.28:0/846732754 wait complete. 2024-09-21T05:46:37.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:37 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/846732754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:37.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:37 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/846732754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:37.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:37 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/846732754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:46:38.100 DEBUG:tasks.fwd_scrub:Yielding 2024-09-21T05:46:38.100 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T05:46:38.114 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T05:46:38.114 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:46:38.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-21T05:46:38.114 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-21T05:46:38.115 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:46:38.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:46:38.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 -- 172.21.15.28:0/3457652940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 msgr2=0x7f52941105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 --2- 172.21.15.28:0/3457652940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f52941105a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f528c009ca0 tx=0x7f528c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 -- 172.21.15.28:0/3457652940 shutdown_connections 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 --2- 172.21.15.28:0/3457652940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f52941105a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 --2- 172.21.15.28:0/3457652940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 0x7f5294109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 --2- 172.21.15.28:0/3457652940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 -- 172.21.15.28:0/3457652940 >> 172.21.15.28:0/3457652940 conn(0x7f52940ff570 msgr2=0x7f52941019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 -- 172.21.15.28:0/3457652940 shutdown_connections 2024-09-21T05:46:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.239+0000 7f529c77b640 1 -- 172.21.15.28:0/3457652940 wait complete. 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 Processor -- start 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 -- start start 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 0x7f5294075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5294113a30 con 0x7f5294109d70 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52941138b0 con 0x7f5294103af0 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529c77b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5294113bb0 con 0x7f5294109420 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529ad18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529ad18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47862/0 (socket says 172.21.15.28:47862) 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f529ad18640 1 -- 172.21.15.28:0/1455175158 learned_addr learned my addr 172.21.15.28:0/1455175158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.240+0000 7f5299d16640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 0x7f5294075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529a517640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 -- 172.21.15.28:0/1455175158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 msgr2=0x7f5294075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 0x7f5294075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 -- 172.21.15.28:0/1455175158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 msgr2=0x7f5294075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 -- 172.21.15.28:0/1455175158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52941a8700 con 0x7f5294109d70 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529a517640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529ad18640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f528c005cc0 tx=0x7f528c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c003bf0 con 0x7f5294109d70 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52941a8990 con 0x7f5294109d70 2024-09-21T05:46:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f528c003d90 con 0x7f5294109d70 2024-09-21T05:46:38.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c02fbc0 con 0x7f5294109d70 2024-09-21T05:46:38.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.241+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52941a8ee0 con 0x7f5294109d70 2024-09-21T05:46:38.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.242+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5294103f00 con 0x7f5294109d70 2024-09-21T05:46:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.245+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f528c02fd60 con 0x7f5294109d70 2024-09-21T05:46:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.246+0000 7f52877fe640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 0x7f52800b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.246+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f528c1089b0 con 0x7f5294109d70 2024-09-21T05:46:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.246+0000 7f529a517640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 0x7f52800b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.247+0000 7f529a517640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 0x7f52800b8aa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5288005fd0 tx=0x7f5288005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.248+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f528c0cf0e0 con 0x7f5294109d70 2024-09-21T05:46:38.391 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.473+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5294063800 con 0x7f52800b6590 2024-09-21T05:46:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.640+0000 7f52877fe640 1 -- 172.21.15.28:0/1455175158 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5294063800 con 0x7f52800b6590 2024-09-21T05:46:38.642 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.642+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 msgr2=0x7f52800b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.642+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 0x7f52800b8aa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5288005fd0 tx=0x7f5288005850 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.642+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 msgr2=0x7f5294076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.642+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f528c005cc0 tx=0x7f528c02f380 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 shutdown_connections 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f52800b6590 0x7f52800b8aa0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5294109d70 0x7f5294076190 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5294109420 0x7f5294075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 --2- 172.21.15.28:0/1455175158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5294103af0 0x7f5294075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 >> 172.21.15.28:0/1455175158 conn(0x7f52940ff570 msgr2=0x7f5294108aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 shutdown_connections 2024-09-21T05:46:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.643+0000 7f529c77b640 1 -- 172.21.15.28:0/1455175158 wait complete. 2024-09-21T05:46:38.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-21T05:46:38.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:38 smithi028 ceph-mon[31227]: pgmap v313: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-09-21T05:46:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 -- 172.21.15.28:0/2005658980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 msgr2=0x7f7690109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 --2- 172.21.15.28:0/2005658980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690109810 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7684009900 tx=0x7f768402a410 comp rx=0 tx=0).stop 2024-09-21T05:46:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 -- 172.21.15.28:0/2005658980 shutdown_connections 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 --2- 172.21.15.28:0/2005658980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 0x7f7690110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 --2- 172.21.15.28:0/2005658980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690109810 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 --2- 172.21.15.28:0/2005658980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 0x7f7690103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 -- 172.21.15.28:0/2005658980 >> 172.21.15.28:0/2005658980 conn(0x7f76900ff510 msgr2=0x7f7690101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 -- 172.21.15.28:0/2005658980 shutdown_connections 2024-09-21T05:46:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.776+0000 7f769565c640 1 -- 172.21.15.28:0/2005658980 wait complete. 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 Processor -- start 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 -- start start 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 0x7f7690075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 0x7f7690076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7690113a10 con 0x7f7690109400 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7690113890 con 0x7f7690109d50 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.777+0000 7f769565c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7690113b90 con 0x7f7690103ad0 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47906/0 (socket says 172.21.15.28:47906) 2024-09-21T05:46:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 -- 172.21.15.28:0/2663327437 learned_addr learned my addr 172.21.15.28:0/2663327437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768effd640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 0x7f7690075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768f7fe640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 0x7f7690076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 -- 172.21.15.28:0/2663327437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 msgr2=0x7f7690075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 0x7f7690075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 -- 172.21.15.28:0/2663327437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 msgr2=0x7f7690076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 0x7f7690076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 -- 172.21.15.28:0/2663327437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76901a8700 con 0x7f7690109400 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f768e7fc640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f768402a930 tx=0x7f7684031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768403f030 con 0x7f7690109400 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76901a8990 con 0x7f7690109400 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7684004490 con 0x7f7690109400 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684035070 con 0x7f7690109400 2024-09-21T05:46:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.778+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76901a8ed0 con 0x7f7690109400 2024-09-21T05:46:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.780+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7690104d40 con 0x7f7690109400 2024-09-21T05:46:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.782+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f7684042030 con 0x7f7690109400 2024-09-21T05:46:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.783+0000 7f7677fff640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 0x7f76780b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.783+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7684107af0 con 0x7f7690109400 2024-09-21T05:46:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.783+0000 7f768effd640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 0x7f76780b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.784+0000 7f768effd640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 0x7f76780b8aa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7690077160 tx=0x7f7680005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.785+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f76840d2660 con 0x7f7690109400 2024-09-21T05:46:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:38 smithi181 ceph-mon[34819]: pgmap v313: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-09-21T05:46:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:38 smithi129 ceph-mon[34885]: pgmap v313: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-09-21T05:46:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 -- 172.21.15.28:0/2296338475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c1052b0 msgr2=0x7f152c10d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 --2- 172.21.15.28:0/2296338475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c1052b0 0x7f152c10d8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f151c009900 tx=0x7f151c02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 -- 172.21.15.28:0/2296338475 shutdown_connections 2024-09-21T05:46:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 --2- 172.21.15.28:0/2296338475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c10de30 0x7f152c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 --2- 172.21.15.28:0/2296338475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c1052b0 0x7f152c10d8f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 --2- 172.21.15.28:0/2296338475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.870+0000 7f1533692640 1 -- 172.21.15.28:0/2296338475 >> 172.21.15.28:0/2296338475 conn(0x7f152c079730 msgr2=0x7f152c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.871+0000 7f1533692640 1 -- 172.21.15.28:0/2296338475 shutdown_connections 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.871+0000 7f1533692640 1 -- 172.21.15.28:0/2296338475 wait complete. 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.871+0000 7f1533692640 1 Processor -- start 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.871+0000 7f1533692640 1 -- start start 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 0x7f152c102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f152c113a70 con 0x7f152c10de30 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1138f0 con 0x7f152c1052b0 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1533692640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f152c113bf0 con 0x7f152c104880 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f153142e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47934/0 (socket says 172.21.15.28:47934) 2024-09-21T05:46:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f153142e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40190/0 (socket says 172.21.15.28:40190) 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 -- 172.21.15.28:0/9471033 learned_addr learned my addr 172.21.15.28:0/9471033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1530c2d640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 0x7f152c102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 -- 172.21.15.28:0/9471033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 msgr2=0x7f152c102790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 -- 172.21.15.28:0/9471033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 msgr2=0x7f152c102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 0x7f152c102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 -- 172.21.15.28:0/9471033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c1a8720 con 0x7f152c10de30 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f153142e640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1530c2d640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 0x7f152c102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.872+0000 7f1531c2f640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f152400eaa0 tx=0x7f152400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.873+0000 7f15227fc640 1 -- 172.21.15.28:0/9471033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152400cdb0 con 0x7f152c10de30 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.873+0000 7f15227fc640 1 -- 172.21.15.28:0/9471033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1524005020 con 0x7f152c10de30 2024-09-21T05:46:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.873+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152c1a8a10 con 0x7f152c10de30 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.873+0000 7f15227fc640 1 -- 172.21.15.28:0/9471033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15240116d0 con 0x7f152c10de30 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.873+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f152c1a8f20 con 0x7f152c10de30 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.876+0000 7f1539c06740 1 Processor -- start 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.877+0000 7f15227fc640 1 -- 172.21.15.28:0/9471033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f15240051d0 con 0x7f152c10de30 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.877+0000 7f1539c06740 1 -- start start 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b795970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b795eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557b6b70be60 con 0x557b6b3ee350 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b6b6fb290 con 0x557b6b3c5490 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1539c06740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557b6b6ff0a0 con 0x557b6b3cd670 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47944/0 (socket says 172.21.15.28:47944) 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 -- 172.21.15.28:0/2769149337 learned_addr learned my addr 172.21.15.28:0/2769149337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f153142e640 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b795970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b795eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 -- 172.21.15.28:0/2769149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 msgr2=0x557b6b795eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b795eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 -- 172.21.15.28:0/2769149337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 msgr2=0x557b6b795970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b795970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 -- 172.21.15.28:0/2769149337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6b459930 con 0x557b6b3ee350 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f15227fc640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 0x7f15180b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f153142e640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 0x7f15180b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.878+0000 7f1530c2d640 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f151c005ab0 tx=0x7f151c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd249f640a29a1d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.879+0000 7f15227fc640 1 -- 172.21.15.28:0/9471033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f15240e7600 con 0x7f152c10de30 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.879+0000 7f15177fe640 1 -- 172.21.15.28:0/2769149337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c035050 con 0x557b6b3ee350 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.879+0000 7f15177fe640 1 -- 172.21.15.28:0/2769149337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f151c0316f0 con 0x557b6b3ee350 2024-09-21T05:46:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.879+0000 7f15177fe640 1 -- 172.21.15.28:0/2769149337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c0046c0 con 0x557b6b3ee350 2024-09-21T05:46:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 -- 172.21.15.28:0/2769149337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 msgr2=0x557b6b795430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f153142e640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 0x7f15180b8af0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x557b6b7940b0 tx=0x7f152800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f151c005ab0 tx=0x7f151c009560 comp rx=0 tx=0).stop 2024-09-21T05:46:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 -- 172.21.15.28:0/2769149337 shutdown_connections 2024-09-21T05:46:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b795eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b795970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 --2- 172.21.15.28:0/2769149337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b795430 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 -- 172.21.15.28:0/2769149337 >> 172.21.15.28:0/2769149337 conn(0x557b6b3e2740 msgr2=0x557b6b793320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 -- 172.21.15.28:0/2769149337 shutdown_connections 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.880+0000 7f1539c06740 1 -- 172.21.15.28:0/2769149337 wait complete. 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 -- start start 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 Processor -- start 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 0x557b6b7a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 0x557b6b7acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557b6b55c890 con 0x557b6b3c5490 2024-09-21T05:46:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b6b695d40 con 0x557b6b3cd670 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557b6b7a3bb0 con 0x557b6b3ee350 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f153142e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f1539c06740 10 client.0 _ll_register_callbacks cb 0x557b6b6f9c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f153142e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47954/0 (socket says 172.21.15.28:47954) 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.881+0000 7f153142e640 1 -- 172.21.15.28:0/980145280 learned_addr learned my addr 172.21.15.28:0/980145280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1530c2d640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 0x557b6b7a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1531c2f640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 0x557b6b7acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 -- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 msgr2=0x557b6b7acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 0x557b6b7acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 -- 172.21.15.28:0/980145280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 msgr2=0x557b6b7a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 0x557b6b7a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 -- 172.21.15.28:0/980145280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b6b7ad4f0 con 0x557b6b3c5490 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1530c2d640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 0x557b6b7a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1531c2f640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 0x557b6b7acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f153142e640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f15280108e0 tx=0x7f1528004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9eb65f4936b4c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152802c060 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1528017030 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.882+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f151c0316f0 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1528010e60 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 10 client.15549 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b6b7afe80 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f151c0316f0 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f152801b390 con 0x557b6b3c5490 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 10 client.15549 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 20 client.15549 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b3accd0 0x557b6b7b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 4 client.15549 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.883+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b6b70ddb0 con 0x557b6b3accd0 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.884+0000 7f1530c2d640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b3accd0 0x557b6b7b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.884+0000 7f1530c2d640 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b3accd0 0x557b6b7b3410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e3682eec0c39dbc server_cookie=aba2a361d6a11d93 in_seq=0 out_seq=0 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.884+0000 7f1514ff9640 10 client.15549 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1514ff9640 1 -- 172.21.15.28:0/980145280 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557b6b70ddb0 con 0x557b6b3accd0 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1514ff9640 10 client.15549 handle_command_reply: tid=0 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 client.15549 shutdown 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 msgr2=0x557b6b7a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f15280108e0 tx=0x7f1528004730 comp rx=0 tx=0).stop 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 shutdown_connections 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b3accd0 0x557b6b7b3410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3ee350 0x557b6b7acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3cd670 0x557b6b7a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 --2- 172.21.15.28:0/980145280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a1070 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 >> 172.21.15.28:0/980145280 conn(0x557b6b3e2740 msgr2=0x557b6b798610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1514ff9640 0 client.15549 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 shutdown_connections 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 1 -- 172.21.15.28:0/980145280 wait complete. 2024-09-21T05:46:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.885+0000 7f1539c06740 20 client.15549 trim_cache size 0 max 16384 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 Processor -- start 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 -- start start 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b79a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b79aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1b17f0 con 0x557b6b3ee350 2024-09-21T05:46:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1b3330 con 0x557b6b3c5490 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1539c06740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1b1ec0 con 0x557b6b3cd670 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1531c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1531c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47966/0 (socket says 172.21.15.28:47966) 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1531c2f640 1 -- 172.21.15.28:0/2870618138 learned_addr learned my addr 172.21.15.28:0/2870618138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f153142e640 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b79a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.924+0000 7f1530c2d640 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b79aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 -- 172.21.15.28:0/2870618138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 msgr2=0x557b6b79aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b79aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 -- 172.21.15.28:0/2870618138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 msgr2=0x557b6b79a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b79a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 -- 172.21.15.28:0/2870618138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6b79c700 con 0x557b6b3ee350 2024-09-21T05:46:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1531c2f640 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f15240aa900 tx=0x7f15240ad1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd58b9b1a2e73f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1516ffd640 1 -- 172.21.15.28:0/2870618138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152401f030 con 0x557b6b3ee350 2024-09-21T05:46:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1516ffd640 1 -- 172.21.15.28:0/2870618138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15240b2170 con 0x557b6b3ee350 2024-09-21T05:46:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.925+0000 7f1516ffd640 1 -- 172.21.15.28:0/2870618138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15240b6040 con 0x557b6b3ee350 2024-09-21T05:46:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- 172.21.15.28:0/2870618138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 msgr2=0x557b6b798530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f15240aa900 tx=0x7f15240ad1a0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- 172.21.15.28:0/2870618138 shutdown_connections 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3ee350 0x557b6b798530 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b79aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 --2- 172.21.15.28:0/2870618138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3c5490 0x557b6b79a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- 172.21.15.28:0/2870618138 >> 172.21.15.28:0/2870618138 conn(0x557b6b3e2740 msgr2=0x557b6b79f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- 172.21.15.28:0/2870618138 shutdown_connections 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- 172.21.15.28:0/2870618138 wait complete. 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.926+0000 7f1539c06740 1 -- start start 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 Processor -- start 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b7a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 0x557b6b81cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1b2c60 con 0x557b6b3c5490 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1b2590 con 0x557b6b3ee350 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f152c1af7d0 con 0x557b6b3cd670 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1539c06740 10 client.0 _ll_register_callbacks cb 0x7f1528010770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f153142e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f1530c2d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b7a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f153142e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47972/0 (socket says 172.21.15.28:47972) 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.927+0000 7f153142e640 1 -- 172.21.15.28:0/3807830182 learned_addr learned my addr 172.21.15.28:0/3807830182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1531c2f640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 0x557b6b81cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 -- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 msgr2=0x557b6b7a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b7a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 -- 172.21.15.28:0/3807830182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 msgr2=0x557b6b81cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 0x557b6b81cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 -- 172.21.15.28:0/3807830182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b6b81d410 con 0x557b6b3c5490 2024-09-21T05:46:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1531c2f640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 0x557b6b81cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f153142e640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f152801b9a0 tx=0x7f152801fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a518a683565864c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152801be30 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1528039480 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.928+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15240b2170 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1528039680 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 10 client.15555 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b6b81e700 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15240b2170 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1528028810 con 0x557b6b3c5490 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 10 client.15555 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 20 client.15555 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b382080 0x557b6b820f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 4 client.15555 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:46:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557b6b709860 con 0x557b6b382080 2024-09-21T05:46:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.929+0000 7f1530c2d640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b382080 0x557b6b820f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.930+0000 7f1530c2d640 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b382080 0x557b6b820f20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d226d5c91ea30f27 server_cookie=99f9d9985c945911 in_seq=0 out_seq=0 2024-09-21T05:46:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.930+0000 7f1514ff9640 10 client.15555 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.931+0000 7f1514ff9640 1 -- 172.21.15.28:0/3807830182 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557b6b709860 con 0x557b6b382080 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.931+0000 7f1514ff9640 10 client.15555 handle_command_reply: tid=0 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.931+0000 7f1539c06740 1 client.15555 shutdown 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 msgr2=0x557b6b7a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f152801b9a0 tx=0x7f152801fd70 comp rx=0 tx=0).stop 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 shutdown_connections 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557b6b382080 0x557b6b820f20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b6b3ee350 0x557b6b81cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557b6b3cd670 0x557b6b7a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 --2- 172.21.15.28:0/3807830182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b6b3c5490 0x557b6b7a56c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 >> 172.21.15.28:0/3807830182 conn(0x557b6b3e2740 msgr2=0x557b6b79f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1514ff9640 0 client.15555 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 shutdown_connections 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 1 -- 172.21.15.28:0/3807830182 wait complete. 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.932+0000 7f1539c06740 20 client.15555 trim_cache size 0 max 16384 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "87d75cd6-0f75-4c86-962f-fa34714c6cd1", 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:46:38.937 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 msgr2=0x7f15180b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 0x7f15180b8af0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x557b6b7940b0 tx=0x7f152800b390 comp rx=0 tx=0).stop 2024-09-21T05:46:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 msgr2=0x7f152c103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f152400eaa0 tx=0x7f152400ef60 comp rx=0 tx=0).stop 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 shutdown_connections 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f15180b65e0 0x7f15180b8af0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f152c10de30 0x7f152c103210 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f152c1052b0 0x7f152c102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 --2- 172.21.15.28:0/9471033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f152c104880 0x7f152c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 >> 172.21.15.28:0/9471033 conn(0x7f152c079730 msgr2=0x7f152c100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 shutdown_connections 2024-09-21T05:46:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:38.933+0000 7f1533692640 1 -- 172.21.15.28:0/9471033 wait complete. 2024-09-21T05:46:39.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.037+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f7690063800 con 0x7f76780b6590 2024-09-21T05:46:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.106+0000 7f7677fff640 1 -- 172.21.15.28:0/2663327437 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f7690063800 con 0x7f76780b6590 2024-09-21T05:46:39.108 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 msgr2=0x7f76780b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 0x7f76780b8aa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7690077160 tx=0x7f7680005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 msgr2=0x7f7690075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f768402a930 tx=0x7f7684031a20 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 shutdown_connections 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7690109d50 0x7f7690076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76780b6590 0x7f76780b8aa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7690109400 0x7f7690075c50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.108+0000 7f769565c640 1 --2- 172.21.15.28:0/2663327437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7690103ad0 0x7f7690075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.109+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 >> 172.21.15.28:0/2663327437 conn(0x7f76900ff510 msgr2=0x7f7690101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.109+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 shutdown_connections 2024-09-21T05:46:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.109+0000 7f769565c640 1 -- 172.21.15.28:0/2663327437 wait complete. 2024-09-21T05:46:39.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T05:46:39.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 -- 172.21.15.28:0/1567773793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c104810 msgr2=0x7f568c104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 --2- 172.21.15.28:0/1567773793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c104810 0x7f568c104c20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5688009900 tx=0x7f568802a410 comp rx=0 tx=0).stop 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 -- 172.21.15.28:0/1567773793 shutdown_connections 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 --2- 172.21.15.28:0/1567773793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c10ddc0 0x7f568c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 --2- 172.21.15.28:0/1567773793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c1051f0 0x7f568c10d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 --2- 172.21.15.28:0/1567773793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c104810 0x7f568c104c20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 -- 172.21.15.28:0/1567773793 >> 172.21.15.28:0/1567773793 conn(0x7f568c0fe3c0 msgr2=0x7f568c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 -- 172.21.15.28:0/1567773793 shutdown_connections 2024-09-21T05:46:39.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.237+0000 7f56933ed640 1 -- 172.21.15.28:0/1567773793 wait complete. 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 Processor -- start 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 -- start start 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c104810 0x7f568c102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c1051f0 0x7f568c102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f568c113a00 con 0x7f568c10ddc0 2024-09-21T05:46:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f568c113880 con 0x7f568c1051f0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.238+0000 7f56933ed640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f568c113b80 con 0x7f568c104810 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47992/0 (socket says 172.21.15.28:47992) 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 -- 172.21.15.28:0/929089716 learned_addr learned my addr 172.21.15.28:0/929089716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f5691189640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c104810 0x7f568c102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f5690988640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c1051f0 0x7f568c102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 -- 172.21.15.28:0/929089716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c104810 msgr2=0x7f568c102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c104810 0x7f568c102720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 -- 172.21.15.28:0/929089716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c1051f0 msgr2=0x7f568c102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c1051f0 0x7f568c102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 -- 172.21.15.28:0/929089716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c1a8710 con 0x7f568c10ddc0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f569198a640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f568400cad0 tx=0x7f568400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568400ae20 con 0x7f568c10ddc0 2024-09-21T05:46:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5684005020 con 0x7f568c10ddc0 2024-09-21T05:46:39.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c1a8a00 con 0x7f568c10ddc0 2024-09-21T05:46:39.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568400f6b0 con 0x7f568c10ddc0 2024-09-21T05:46:39.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.239+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c1a8f50 con 0x7f568c10ddc0 2024-09-21T05:46:39.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.240+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568c104c20 con 0x7f568c10ddc0 2024-09-21T05:46:39.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.243+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 154529+0+0 (secure 0 0 0) 0x7f56840051d0 con 0x7f568c10ddc0 2024-09-21T05:46:39.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.244+0000 7f56827fc640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 0x7f56780b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.244+0000 7f5691189640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 0x7f56780b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.244+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f56840e77d0 con 0x7f568c10ddc0 2024-09-21T05:46:39.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.245+0000 7f5691189640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 0x7f56780b8b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:39.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.247+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5684014030 con 0x7f568c10ddc0 2024-09-21T05:46:39.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.458+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f568c106f30 con 0x7f56780b6610 2024-09-21T05:46:39.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.465+0000 7f56827fc640 1 -- 172.21.15.28:0/929089716 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f568c106f30 con 0x7f56780b6610 2024-09-21T05:46:39.468 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T05:46:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 msgr2=0x7f56780b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 0x7f56780b8b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688031400 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 msgr2=0x7f568c1031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f568400cad0 tx=0x7f568400cf90 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 shutdown_connections 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f56780b6610 0x7f56780b8b20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c10ddc0 0x7f568c1031a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f568c1051f0 0x7f568c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 --2- 172.21.15.28:0/929089716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f568c104810 0x7f568c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 >> 172.21.15.28:0/929089716 conn(0x7f568c0fe3c0 msgr2=0x7f568c106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 shutdown_connections 2024-09-21T05:46:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.468+0000 7f56933ed640 1 -- 172.21.15.28:0/929089716 wait complete. 2024-09-21T05:46:39.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.599+0000 7f2d54750640 1 -- 172.21.15.28:0/2348634765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109430 msgr2=0x7f2d4c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.599+0000 7f2d54750640 1 --2- 172.21.15.28:0/2348634765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109430 0x7f2d4c109840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009900 tx=0x7f2d3c02a410 comp rx=0 tx=0).stop 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 -- 172.21.15.28:0/2348634765 shutdown_connections 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 --2- 172.21.15.28:0/2348634765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c109d80 0x7f2d4c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 --2- 172.21.15.28:0/2348634765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109430 0x7f2d4c109840 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 --2- 172.21.15.28:0/2348634765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c103b00 0x7f2d4c103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 -- 172.21.15.28:0/2348634765 >> 172.21.15.28:0/2348634765 conn(0x7f2d4c0ff500 msgr2=0x7f2d4c101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 -- 172.21.15.28:0/2348634765 shutdown_connections 2024-09-21T05:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 -- 172.21.15.28:0/2348634765 wait complete. 2024-09-21T05:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 Processor -- start 2024-09-21T05:46:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.600+0000 7f2d54750640 1 -- start start 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c103b00 0x7f2d4c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 0x7f2d4c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c113a90 con 0x7f2d4c109d80 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c113910 con 0x7f2d4c109430 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d54750640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c113c10 con 0x7f2d4c103b00 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48008/0 (socket says 172.21.15.28:48008) 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 -- 172.21.15.28:0/3112835643 learned_addr learned my addr 172.21.15.28:0/3112835643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d524ec640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c103b00 0x7f2d4c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d51ceb640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 0x7f2d4c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 -- 172.21.15.28:0/3112835643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c103b00 msgr2=0x7f2d4c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c103b00 0x7f2d4c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 -- 172.21.15.28:0/3112835643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 msgr2=0x7f2d4c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 0x7f2d4c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 -- 172.21.15.28:0/3112835643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c1a03f0 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d51ceb640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 0x7f2d4c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d52ced640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400cce0 tx=0x7f2d4400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.601+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400f450 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.602+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d4400faa0 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.602+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c1a06e0 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.602+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44018b70 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.602+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c1a0be0 con 0x7f2d4c109d80 2024-09-21T05:46:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.603+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d4c103f10 con 0x7f2d4c109d80 2024-09-21T05:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.604+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 154586+0+0 (secure 0 0 0) 0x7f2d44018d10 con 0x7f2d4c109d80 2024-09-21T05:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.605+0000 7f2d437fe640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 0x7f2d380b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.605+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f2d440e7b20 con 0x7f2d4c109d80 2024-09-21T05:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.605+0000 7f2d524ec640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 0x7f2d380b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.606+0000 7f2d524ec640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 0x7f2d380b8b00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009bd0 tx=0x7f2d48005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.608+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2d44014030 con 0x7f2d4c109d80 2024-09-21T05:46:39.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:46:39.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:39 smithi181 ceph-mon[34819]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:39.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:39 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:46:39.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:39 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:39 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:46:39.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:39 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:39 smithi028 ceph-mon[31227]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:39 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:46:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:39 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:39 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:46:39.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:39 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.826+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2d4c063800 con 0x7f2d380b65f0 2024-09-21T05:46:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.834+0000 7f2d437fe640 1 -- 172.21.15.28:0/3112835643 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2d4c063800 con 0x7f2d380b65f0 2024-09-21T05:46:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 msgr2=0x7f2d380b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 0x7f2d380b8b00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009bd0 tx=0x7f2d48005f90 comp rx=0 tx=0).stop 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 msgr2=0x7f2d4c19fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400cce0 tx=0x7f2d4400a620 comp rx=0 tx=0).stop 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 shutdown_connections 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2d380b65f0 0x7f2d380b8b00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c109d80 0x7f2d4c19fd80 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4c109430 0x7f2d4c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 --2- 172.21.15.28:0/3112835643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4c103b00 0x7f2d4c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 >> 172.21.15.28:0/3112835643 conn(0x7f2d4c0ff500 msgr2=0x7f2d4c106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 shutdown_connections 2024-09-21T05:46:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:39.837+0000 7f2d54750640 1 -- 172.21.15.28:0/3112835643 wait complete. 2024-09-21T05:46:39.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-21T05:46:39.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:39 smithi129 ceph-mon[34885]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:39.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:39 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:46:39.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:39 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:39 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:46:39.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:39 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:46:39.880 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-21T05:46:39.891 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-21T05:46:39.913 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:46:39.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-21T05:46:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:46:39.920 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T05:46:39.942 INFO:tasks.workunit:Pulling workunits from ref 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:46:39.943 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T05:46:39.943 INFO:tasks.workunit:timeout=3h 2024-09-21T05:46:39.943 INFO:tasks.workunit:cleanup=True 2024-09-21T05:46:39.943 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-21 05:44:58.395561759 +0000 2024-09-21T05:46:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-21 05:44:58.395561759 +0000 2024-09-21T05:46:39.961 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-21 05:46:29.373269758 +0000 2024-09-21T05:46:39.961 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-21 05:44:58.395561759 +0000 2024-09-21T05:46:39.961 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-21T05:46:39.961 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-21T05:46:40.036 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T05:46:40.092 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 -- 172.21.15.28:0/3397986619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c98109e00 msgr2=0x7f4c98110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/3397986619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c98109e00 0x7f4c98110630 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009ca0 tx=0x7f4c9002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 -- 172.21.15.28:0/3397986619 shutdown_connections 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/3397986619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c98109e00 0x7f4c98110630 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/3397986619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c981094b0 0x7f4c981098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/3397986619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 0x7f4c98103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 -- 172.21.15.28:0/3397986619 >> 172.21.15.28:0/3397986619 conn(0x7f4c9807b910 msgr2=0x7f4c9807bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 -- 172.21.15.28:0/3397986619 shutdown_connections 2024-09-21T05:46:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.445+0000 7f4c9e18d640 1 -- 172.21.15.28:0/3397986619 wait complete. 2024-09-21T05:46:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 Processor -- start 2024-09-21T05:46:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 -- start start 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 0x7f4c98075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c98109e00 0x7f4c98076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98113ac0 con 0x7f4c981094b0 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98113940 con 0x7f4c98103bc0 2024-09-21T05:46:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c9e18d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98113c40 con 0x7f4c98109e00 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48032/0 (socket says 172.21.15.28:48032) 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.446+0000 7f4c96ffd640 1 -- 172.21.15.28:0/1090606865 learned_addr learned my addr 172.21.15.28:0/1090606865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c977fe640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 0x7f4c98075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c97fff640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c98109e00 0x7f4c98076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 -- 172.21.15.28:0/1090606865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c98109e00 msgr2=0x7f4c98076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c98109e00 0x7f4c98076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 -- 172.21.15.28:0/1090606865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 msgr2=0x7f4c98075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 0x7f4c98075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 -- 172.21.15.28:0/1090606865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c981a87b0 con 0x7f4c981094b0 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000c970 tx=0x7f4c8000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c94ff9640 1 -- 172.21.15.28:0/1090606865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80004220 con 0x7f4c981094b0 2024-09-21T05:46:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c981a8aa0 con 0x7f4c981094b0 2024-09-21T05:46:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c94ff9640 1 -- 172.21.15.28:0/1090606865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c800043c0 con 0x7f4c981094b0 2024-09-21T05:46:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c94ff9640 1 -- 172.21.15.28:0/1090606865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80004a00 con 0x7f4c981094b0 2024-09-21T05:46:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.447+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981a8f90 con 0x7f4c981094b0 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.450+0000 7f4c94ff9640 1 -- 172.21.15.28:0/1090606865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 154586+0+0 (secure 0 0 0) 0x7f4c800079b0 con 0x7f4c981094b0 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.450+0000 7f4ca4701740 1 Processor -- start 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.450+0000 7f4ca4701740 1 -- start start 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x558882675b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ad390 0x5588826760d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5588825ebe50 con 0x5588822ce070 2024-09-21T05:46:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5588825db280 con 0x5588822ad390 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4ca4701740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5588825df090 con 0x5588822a51b0 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48044/0 (socket says 172.21.15.28:48044) 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 -- 172.21.15.28:0/136204901 learned_addr learned my addr 172.21.15.28:0/136204901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c977fe640 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x558882675b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ad390 0x5588826760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 -- 172.21.15.28:0/136204901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 msgr2=0x558882675b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x558882675b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 -- 172.21.15.28:0/136204901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ad390 msgr2=0x5588826760d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ad390 0x5588826760d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c97fff640 1 -- 172.21.15.28:0/136204901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558882339650 con 0x5588822ce070 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c94ff9640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 0x7f4c840b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.451+0000 7f4c977fe640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 0x7f4c840b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.452+0000 7f4c97fff640 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009930 tx=0x7f4c90009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8596f672ddc32fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.452+0000 7f4c94ff9640 1 -- 172.21.15.28:0/1090606865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4c800e73d0 con 0x7f4c981094b0 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.452+0000 7f4c8dffb640 1 -- 172.21.15.28:0/136204901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90031df0 con 0x5588822ce070 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.452+0000 7f4c8dffb640 1 -- 172.21.15.28:0/136204901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90036040 con 0x5588822ce070 2024-09-21T05:46:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.452+0000 7f4c8dffb640 1 -- 172.21.15.28:0/136204901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c900042a0 con 0x5588822ce070 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- 172.21.15.28:0/136204901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 msgr2=0x558882675650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4c977fe640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 0x7f4c840b8b00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5588826742d0 tx=0x7f4c8800c480 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009930 tx=0x7f4c90009960 comp rx=0 tx=0).stop 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- 172.21.15.28:0/136204901 shutdown_connections 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ad390 0x5588826760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x558882675b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 --2- 172.21.15.28:0/136204901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ce070 0x558882675650 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- 172.21.15.28:0/136204901 >> 172.21.15.28:0/136204901 conn(0x5588822c2460 msgr2=0x558882673540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- 172.21.15.28:0/136204901 shutdown_connections 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- 172.21.15.28:0/136204901 wait complete. 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.453+0000 7f4ca4701740 1 -- start start 2024-09-21T05:46:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 Processor -- start 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822ad390 0x558882681270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ce070 0x55888268d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55888243c700 con 0x5588822a51b0 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558882576110 con 0x5588822ce070 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558882683dd0 con 0x5588822ad390 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4ca4701740 10 client.0 _ll_register_callbacks cb 0x55888258f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48048/0 (socket says 172.21.15.28:48048) 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 -- 172.21.15.28:0/701413251 learned_addr learned my addr 172.21.15.28:0/701413251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822ad390 0x558882681270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c97fff640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ce070 0x55888268d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 -- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822ad390 msgr2=0x558882681270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822ad390 0x558882681270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 -- 172.21.15.28:0/701413251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ce070 msgr2=0x55888268d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ce070 0x55888268d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.454+0000 7f4c977fe640 1 -- 172.21.15.28:0/701413251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588825a3c40 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c977fe640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c8801cdf0 tx=0x7f4c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f19787d661c725 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88029060 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4c8800cd70 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c90036040 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c8801a460 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 10 client.15579 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55888268fb30 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90036040 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4c88018040 con 0x5588822a51b0 2024-09-21T05:46:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 10 client.15579 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 20 client.15579 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55888228c9f0 0x558882693090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 4 client.15579 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.455+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588825edda0 con 0x55888228c9f0 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.456+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55888228c9f0 0x558882693090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.456+0000 7f4c76ffd640 10 client.15579 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.456+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55888228c9f0 0x558882693090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa7a00e9d5bfd4b4 server_cookie=25da9b92cecd99f9 in_seq=0 out_seq=0 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4c76ffd640 1 -- 172.21.15.28:0/701413251 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5588825edda0 con 0x55888228c9f0 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4c76ffd640 10 client.15579 handle_command_reply: tid=0 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 client.15579 shutdown 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 msgr2=0x558882680d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c8801cdf0 tx=0x7f4c880046c0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 shutdown_connections 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55888228c9f0 0x558882693090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4c76ffd640 0 client.15579 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822ce070 0x55888268d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822ad390 0x558882681270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 --2- 172.21.15.28:0/701413251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822a51b0 0x558882680d30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.457+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 >> 172.21.15.28:0/701413251 conn(0x5588822c2460 msgr2=0x558882678830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.458+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 shutdown_connections 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.458+0000 7f4ca4701740 1 -- 172.21.15.28:0/701413251 wait complete. 2024-09-21T05:46:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.458+0000 7f4ca4701740 20 client.15579 trim_cache size 0 max 16384 2024-09-21T05:46:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.475+0000 7f4ca4701740 1 Processor -- start 2024-09-21T05:46:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.475+0000 7f4ca4701740 1 -- start start 2024-09-21T05:46:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55888228c9f0 0x5588826811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x55888267df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981b1a00 con 0x5588822ad390 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981b3540 con 0x55888228c9f0 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4ca4701740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981b20d0 con 0x5588822a51b0 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48052/0 (socket says 172.21.15.28:48052) 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 -- 172.21.15.28:0/341705083 learned_addr learned my addr 172.21.15.28:0/341705083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x55888267df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c977fe640 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55888228c9f0 0x5588826811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 -- 172.21.15.28:0/341705083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 msgr2=0x55888267df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x55888267df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 -- 172.21.15.28:0/341705083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55888228c9f0 msgr2=0x5588826811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55888228c9f0 0x5588826811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 -- 172.21.15.28:0/341705083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55888267fc90 con 0x5588822ad390 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c97fff640 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006010 tx=0x7f4c9002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe7b3b1c9ee69d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c8d7fa640 1 -- 172.21.15.28:0/341705083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90052070 con 0x5588822ad390 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.476+0000 7f4c8d7fa640 1 -- 172.21.15.28:0/341705083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90036070 con 0x5588822ad390 2024-09-21T05:46:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4c8d7fa640 1 -- 172.21.15.28:0/341705083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c900042a0 con 0x5588822ad390 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 -- 172.21.15.28:0/341705083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 msgr2=0x55888267e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006010 tx=0x7f4c9002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 -- 172.21.15.28:0/341705083 shutdown_connections 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x55888267e460 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5588822a51b0 0x55888267df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 --2- 172.21.15.28:0/341705083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55888228c9f0 0x5588826811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 -- 172.21.15.28:0/341705083 >> 172.21.15.28:0/341705083 conn(0x5588822c2460 msgr2=0x558882680ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 -- 172.21.15.28:0/341705083 shutdown_connections 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.477+0000 7f4ca4701740 1 -- 172.21.15.28:0/341705083 wait complete. 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 -- start start 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 Processor -- start 2024-09-21T05:46:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 0x558882685310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822a51b0 0x558882685850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981b2e70 con 0x5588822ad390 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981b27a0 con 0x5588822a51b0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981af9e0 con 0x55888228c9f0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.478+0000 7f4ca4701740 10 client.0 _ll_register_callbacks cb 0x7f4c8800c9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 0x558882685310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48058/0 (socket says 172.21.15.28:48058) 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 0x558882685310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40294/0 (socket says 172.21.15.28:40294) 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 -- 172.21.15.28:0/3393647478 learned_addr learned my addr 172.21.15.28:0/3393647478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c96ffd640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822a51b0 0x558882685850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 -- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 msgr2=0x558882685310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 0x558882685310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 -- 172.21.15.28:0/3393647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822a51b0 msgr2=0x558882685850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822a51b0 0x558882685850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 -- 172.21.15.28:0/3393647478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588826860d0 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c97fff640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4c90050040 tx=0x7f4c9002f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988ad557234fd9cf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90052070 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c90036070 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.479+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4c90031450 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c90031650 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 10 client.15585 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588826fdfe0 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90036070 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4c90030650 con 0x5588822ad390 2024-09-21T05:46:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 10 client.15585 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 20 client.15585 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5588822ce070 0x5588827002c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 4 client.15585 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588825e9850 con 0x5588822ce070 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.480+0000 7f4c977fe640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5588822ce070 0x5588827002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4c977fe640 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5588822ce070 0x5588827002c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24dbade6917de13c server_cookie=c79fc7fb7788de74 in_seq=0 out_seq=0 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4c76ffd640 10 client.15585 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4c76ffd640 1 -- 172.21.15.28:0/3393647478 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+570 (crc 0 0 0) 0x5588825e9850 con 0x5588822ce070 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4c76ffd640 10 client.15585 handle_command_reply: tid=0 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 client.15585 shutdown 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 msgr2=0x5588826fca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4c90050040 tx=0x7f4c9002f420 comp rx=0 tx=0).stop 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 shutdown_connections 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5588822ce070 0x5588827002c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5588822ad390 0x5588826fca50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5588822a51b0 0x558882685850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 --2- 172.21.15.28:0/3393647478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55888228c9f0 0x558882685310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4c76ffd640 0 client.15585 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 >> 172.21.15.28:0/3393647478 conn(0x5588822c2460 msgr2=0x558882676680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 shutdown_connections 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 1 -- 172.21.15.28:0/3393647478 wait complete. 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.481+0000 7f4ca4701740 20 client.15585 trim_cache size 0 max 16384 2024-09-21T05:46:40.489 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "87d75cd6-0f75-4c86-962f-fa34714c6cd1", 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:46:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:46:40.491 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 msgr2=0x7f4c840b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 0x7f4c840b8b00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5588826742d0 tx=0x7f4c8800c480 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 msgr2=0x7f4c98075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000c970 tx=0x7f4c8000ce30 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 shutdown_connections 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c98109e00 0x7f4c98076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4c840b65f0 0x7f4c840b8b00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c981094b0 0x7f4c98075c50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 --2- 172.21.15.28:0/1090606865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c98103bc0 0x7f4c98075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 >> 172.21.15.28:0/1090606865 conn(0x7f4c9807b910 msgr2=0x7f4c98107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 shutdown_connections 2024-09-21T05:46:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:46:40.482+0000 7f4c9e18d640 1 -- 172.21.15.28:0/1090606865 wait complete. 2024-09-21T05:46:40.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:40 smithi028 ceph-mon[31227]: pgmap v314: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-21T05:46:40.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:40 smithi028 ceph-mon[31227]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:40 smithi028 ceph-mon[31227]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:40 smithi028 ceph-mon[31227]: mgrmap e24: x(active, since 8m), standbys: z, y 2024-09-21T05:46:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:40 smithi181 ceph-mon[34819]: pgmap v314: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-21T05:46:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:40 smithi181 ceph-mon[34819]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:40 smithi181 ceph-mon[34819]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:40 smithi181 ceph-mon[34819]: mgrmap e24: x(active, since 8m), standbys: z, y 2024-09-21T05:46:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:40 smithi129 ceph-mon[34885]: pgmap v314: 97 pgs: 97 active+clean; 653 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-21T05:46:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:40 smithi129 ceph-mon[34885]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:40 smithi129 ceph-mon[34885]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:40 smithi129 ceph-mon[34885]: mgrmap e24: x(active, since 8m), standbys: z, y 2024-09-21T05:46:41.278 INFO:tasks.cephfs.filesystem:scrub status for tag:87d75cd6-0f75-4c86-962f-fa34714c6cd1 - {'path': '/', 'tag': '87d75cd6-0f75-4c86-962f-fa34714c6cd1', 'options': 'recursive,force'} 2024-09-21T05:46:41.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:41 smithi181 ceph-mon[34819]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:41.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:41 smithi028 ceph-mon[31227]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:41.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:41 smithi129 ceph-mon[34885]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:46:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:42 smithi181 ceph-mon[34819]: pgmap v315: 97 pgs: 97 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 91 KiB/s wr, 32 op/s 2024-09-21T05:46:42.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:42 smithi028 ceph-mon[31227]: pgmap v315: 97 pgs: 97 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 91 KiB/s wr, 32 op/s 2024-09-21T05:46:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:42 smithi129 ceph-mon[34885]: pgmap v315: 97 pgs: 97 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 91 KiB/s wr, 32 op/s 2024-09-21T05:46:44.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:44 smithi181 ceph-mon[34819]: pgmap v316: 97 pgs: 97 active+clean; 1.4 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 165 KiB/s wr, 43 op/s 2024-09-21T05:46:44.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:44 smithi028 ceph-mon[31227]: pgmap v316: 97 pgs: 97 active+clean; 1.4 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 165 KiB/s wr, 43 op/s 2024-09-21T05:46:44.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:44 smithi129 ceph-mon[34885]: pgmap v316: 97 pgs: 97 active+clean; 1.4 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 165 KiB/s wr, 43 op/s 2024-09-21T05:46:45.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:45 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:46:45.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:45 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:46:45.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:45 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:46:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:46 smithi181 ceph-mon[34819]: pgmap v317: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:46 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:46 smithi028 ceph-mon[31227]: pgmap v317: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:46.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:46 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:46.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:46 smithi129 ceph-mon[34885]: pgmap v317: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:46.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:46.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:46 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: pgmap v318: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:47.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:46:47.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:46:47.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: pgmap v318: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:46:47.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:48.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: pgmap v318: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:46:48.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:46:50.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:49 smithi028 ceph-mon[31227]: pgmap v319: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T05:46:50.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:49 smithi129 ceph-mon[34885]: pgmap v319: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T05:46:50.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:49 smithi181 ceph-mon[34819]: pgmap v319: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T05:46:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:51 smithi028 ceph-mon[31227]: pgmap v320: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:51 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:46:52.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:51 smithi129 ceph-mon[34885]: pgmap v320: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:52.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:51 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:46:52.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:51 smithi181 ceph-mon[34819]: pgmap v320: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-09-21T05:46:52.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:51 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 -- 172.21.15.28:0/2494315504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 msgr2=0x7f5424107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 --2- 172.21.15.28:0/2494315504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f5424107ed0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5414009900 tx=0x7f541402a410 comp rx=0 tx=0).stop 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 -- 172.21.15.28:0/2494315504 shutdown_connections 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 --2- 172.21.15.28:0/2494315504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 0x7f542410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 --2- 172.21.15.28:0/2494315504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f5424107ed0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 --2- 172.21.15.28:0/2494315504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 0x7f54241025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 -- 172.21.15.28:0/2494315504 >> 172.21.15.28:0/2494315504 conn(0x7f5424079d00 msgr2=0x7f542407a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 -- 172.21.15.28:0/2494315504 shutdown_connections 2024-09-21T05:46:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 -- 172.21.15.28:0/2494315504 wait complete. 2024-09-21T05:46:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.076+0000 7f542c7a1640 1 Processor -- start 2024-09-21T05:46:53.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 -- start start 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 0x7f542419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 0x7f542419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5424113a40 con 0x7f5424107ac0 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54241138c0 con 0x7f5424108410 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542c7a1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5424113bc0 con 0x7f5424102190 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60202/0 (socket says 172.21.15.28:60202) 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 -- 172.21.15.28:0/1646581888 learned_addr learned my addr 172.21.15.28:0/1646581888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542a53d640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 0x7f542419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f542ad3e640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 0x7f542419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 -- 172.21.15.28:0/1646581888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 msgr2=0x7f542419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 0x7f542419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 -- 172.21.15.28:0/1646581888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 msgr2=0x7f542419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 0x7f542419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.077+0000 7f5429d3c640 1 -- 172.21.15.28:0/1646581888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54241a03b0 con 0x7f5424107ac0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f5429d3c640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f541402a930 tx=0x7f5414031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f541b7fe640 1 -- 172.21.15.28:0/1646581888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541403f030 con 0x7f5424107ac0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f541b7fe640 1 -- 172.21.15.28:0/1646581888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5414004490 con 0x7f5424107ac0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54241a0640 con 0x7f5424107ac0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f541b7fe640 1 -- 172.21.15.28:0/1646581888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414035070 con 0x7f5424107ac0 2024-09-21T05:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.078+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54241a6e30 con 0x7f5424107ac0 2024-09-21T05:46:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.080+0000 7f54337ea740 1 Processor -- start 2024-09-21T05:46:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 -- start start 2024-09-21T05:46:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b32430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e11aa66f0 con 0x564e117632f0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564e11a95b20 con 0x564e1176b4d0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f54337ea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564e11a99930 con 0x564e1178c1b0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f541b7fe640 1 -- 172.21.15.28:0/1646581888 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 154586+0+0 (secure 0 0 0) 0x7f5414042030 con 0x7f5424107ac0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f542a53d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f542a53d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60212/0 (socket says 172.21.15.28:60212) 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f542a53d640 1 -- 172.21.15.28:0/4232516408 learned_addr learned my addr 172.21.15.28:0/4232516408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.081+0000 7f542ad3e640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f5429d3c640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b32430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 -- 172.21.15.28:0/4232516408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 msgr2=0x564e11b319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 -- 172.21.15.28:0/4232516408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 msgr2=0x564e11b32430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b32430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 -- 172.21.15.28:0/4232516408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e11a5b530 con 0x564e117632f0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f5429d3c640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b32430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542ad3e640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542a53d640 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5420009cf0 tx=0x7f5420011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70706f58097bf304 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f5418ff9640 1 -- 172.21.15.28:0/4232516408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420004230 con 0x564e117632f0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f5418ff9640 1 -- 172.21.15.28:0/4232516408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54200043d0 con 0x564e117632f0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f5418ff9640 1 -- 172.21.15.28:0/4232516408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420019880 con 0x564e117632f0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f541b7fe640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 0x7f54100b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f542ad3e640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 0x7f54100b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.082+0000 7f541b7fe640 1 -- 172.21.15.28:0/1646581888 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5414107540 con 0x7f5424107ac0 2024-09-21T05:46:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 -- 172.21.15.28:0/4232516408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 msgr2=0x564e11b31ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f542ad3e640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 0x7f54100b8b00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fd0 tx=0x7f541c009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5420009cf0 tx=0x7f5420011810 comp rx=0 tx=0).stop 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 -- 172.21.15.28:0/4232516408 shutdown_connections 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b32430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b31ef0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 --2- 172.21.15.28:0/4232516408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 -- 172.21.15.28:0/4232516408 >> 172.21.15.28:0/4232516408 conn(0x564e11780300 msgr2=0x564e11b2f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 -- 172.21.15.28:0/4232516408 shutdown_connections 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.083+0000 7f54337ea740 1 -- 172.21.15.28:0/4232516408 wait complete. 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 -- start start 2024-09-21T05:46:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 Processor -- start 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b41ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e119cdab0 con 0x564e1176b4d0 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564e11a3b730 con 0x564e117632f0 2024-09-21T05:46:53.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564e11b44510 con 0x564e1178c1b0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f54337ea740 10 client.0 _ll_register_callbacks cb 0x564e11a8f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60228/0 (socket says 172.21.15.28:60228) 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f5429d3c640 1 -- 172.21.15.28:0/3974876634 learned_addr learned my addr 172.21.15.28:0/3974876634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.084+0000 7f542ad3e640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b41ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 -- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 msgr2=0x564e11b41ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b41ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 -- 172.21.15.28:0/3974876634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 msgr2=0x564e11b413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f542a53d640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 -- 172.21.15.28:0/3974876634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e117f7790 con 0x564e1176b4d0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f5429d3c640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5414031440 tx=0x7f5414003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4117e11409159526 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414049040 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f54140d6c00 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54200043d0 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f54140ccc90 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f54337ea740 10 client.15597 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e11870620 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54200043d0 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.085+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f54140b4070 con 0x564e1176b4d0 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 10 client.15597 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 20 client.15597 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e1174ad80 0x564e11b534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 4 client.15597 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e11aa8640 con 0x564e1174ad80 2024-09-21T05:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f542a53d640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e1174ad80 0x564e11b534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f542a53d640 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e1174ad80 0x564e11b534e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8c65dca6b5886c2 server_cookie=ab8ed758af7f463f in_seq=0 out_seq=0 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.086+0000 7f540dffb640 10 client.15597 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f540dffb640 1 -- 172.21.15.28:0/3974876634 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564e11aa8640 con 0x564e1174ad80 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f540dffb640 10 client.15597 handle_command_reply: tid=0 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 client.15597 shutdown 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 msgr2=0x564e11b41920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5414031440 tx=0x7f5414003f60 comp rx=0 tx=0).stop 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 shutdown_connections 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e1174ad80 0x564e11b534e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b41ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b41920 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 --2- 172.21.15.28:0/3974876634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f540dffb640 0 client.15597 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 >> 172.21.15.28:0/3974876634 conn(0x564e11780300 msgr2=0x564e11b333a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 shutdown_connections 2024-09-21T05:46:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.087+0000 7f54337ea740 1 -- 172.21.15.28:0/3974876634 wait complete. 2024-09-21T05:46:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.088+0000 7f54337ea740 20 client.15597 trim_cache size 0 max 16384 2024-09-21T05:46:53.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.102+0000 7f54337ea740 1 Processor -- start 2024-09-21T05:46:53.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.102+0000 7f54337ea740 1 -- start start 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b3e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b3eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ad4e0 con 0x564e117632f0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54241af020 con 0x564e1176b4d0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f54337ea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54241adbb0 con 0x564e1178c1b0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60240/0 (socket says 172.21.15.28:60240) 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 -- 172.21.15.28:0/3022412781 learned_addr learned my addr 172.21.15.28:0/3022412781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542ad3e640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b3eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f5429d3c640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b3e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 -- 172.21.15.28:0/3022412781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 msgr2=0x564e11b3eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b3eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 -- 172.21.15.28:0/3022412781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 msgr2=0x564e11b3e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b3e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 -- 172.21.15.28:0/3022412781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e11b3f8c0 con 0x564e117632f0 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f5429d3c640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b3e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f542a53d640 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5420009a10 tx=0x7f5420003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78276bf4e7a55750 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.103+0000 7f540ffff640 1 -- 172.21.15.28:0/3022412781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420005040 con 0x564e117632f0 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f540ffff640 1 -- 172.21.15.28:0/3022412781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54200035f0 con 0x564e117632f0 2024-09-21T05:46:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f540ffff640 1 -- 172.21.15.28:0/3022412781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200197b0 con 0x564e117632f0 2024-09-21T05:46:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 -- 172.21.15.28:0/3022412781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 msgr2=0x564e11b2a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5420009a10 tx=0x7f5420003f10 comp rx=0 tx=0).stop 2024-09-21T05:46:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 -- 172.21.15.28:0/3022412781 shutdown_connections 2024-09-21T05:46:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b3eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e1176b4d0 0x564e11b3e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 --2- 172.21.15.28:0/3022412781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e117632f0 0x564e11b2a5c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 -- 172.21.15.28:0/3022412781 >> 172.21.15.28:0/3022412781 conn(0x564e11780300 msgr2=0x564e11b3fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 -- 172.21.15.28:0/3022412781 shutdown_connections 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.104+0000 7f54337ea740 1 -- 172.21.15.28:0/3022412781 wait complete. 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 -- start start 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 Processor -- start 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b45710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b46190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ae950 con 0x564e1176b4d0 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ae280 con 0x564e117632f0 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ab4c0 con 0x564e1178c1b0 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f54337ea740 10 client.0 _ll_register_callbacks cb 0x7f5414005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:46:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60252/0 (socket says 172.21.15.28:60252) 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 -- 172.21.15.28:0/2419414209 learned_addr learned my addr 172.21.15.28:0/2419414209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f542a53d640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b45710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f542ad3e640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b46190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 -- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 msgr2=0x564e11b46190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b46190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 -- 172.21.15.28:0/2419414209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 msgr2=0x564e11b45710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b45710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.105+0000 7f5429d3c640 1 -- 172.21.15.28:0/2419414209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e11bbcee0 con 0x564e1176b4d0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f5429d3c640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f54140ca790 tx=0x7f54140cca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed194ac67a96f638 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414049040 con 0x564e1176b4d0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5414031ac0 con 0x564e1176b4d0 2024-09-21T05:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54200035f0 con 0x564e1176b4d0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f54140c9b70 con 0x564e1176b4d0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 10 client.15603 fetch_fsmap learned FSMap version 27 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e11bbe050 con 0x564e1176b4d0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54200035f0 con 0x564e1176b4d0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f54140b4070 con 0x564e1176b4d0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 10 client.15603 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 20 client.15603 populate_metadata read hostname 'smithi028' 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e117201c0 0x564e11bc0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 4 client.15603 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.106+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e11aa40f0 con 0x564e117201c0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.107+0000 7f542a53d640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e117201c0 0x564e11bc0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.107+0000 7f540dffb640 10 client.15603 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.107+0000 7f542a53d640 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e117201c0 0x564e11bc0680 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eab4a5fa49417de server_cookie=98782d1191d0ed7 in_seq=0 out_seq=0 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.118+0000 7f540dffb640 1 -- 172.21.15.28:0/2419414209 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5420034040 con 0x564e117201c0 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.118+0000 7f540dffb640 10 client.15603 handle_command_reply: tid=0 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.118+0000 7f54337ea740 1 client.15603 shutdown 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.118+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 msgr2=0x564e11b45c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.118+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f54140ca790 tx=0x7f54140cca30 comp rx=0 tx=0).stop 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 shutdown_connections 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564e117201c0 0x564e11bc0680 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e1178c1b0 0x564e11b46190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e1176b4d0 0x564e11b45c50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 --2- 172.21.15.28:0/2419414209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564e117632f0 0x564e11b45710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f540dffb640 0 client.15603 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 >> 172.21.15.28:0/2419414209 conn(0x564e11780300 msgr2=0x564e11b2dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 shutdown_connections 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 1 -- 172.21.15.28:0/2419414209 wait complete. 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f54337ea740 20 client.15603 trim_cache size 0 max 16384 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:46:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 msgr2=0x7f54100b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 0x7f54100b8b00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fd0 tx=0x7f541c009500 comp rx=0 tx=0).stop 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 msgr2=0x7f542419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f541402a930 tx=0x7f5414031a20 comp rx=0 tx=0).stop 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 shutdown_connections 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5424108410 0x7f542419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f54100b65f0 0x7f54100b8b00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424107ac0 0x7f542419aa20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 --2- 172.21.15.28:0/1646581888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5424102190 0x7f542419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:46:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 >> 172.21.15.28:0/1646581888 conn(0x7f5424079d00 msgr2=0x7f5424103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:46:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.119+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 shutdown_connections 2024-09-21T05:46:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:46:53.120+0000 7f542c7a1640 1 -- 172.21.15.28:0/1646581888 wait complete. 2024-09-21T05:46:54.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:53 smithi028 ceph-mon[31227]: pgmap v321: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 84 KiB/s wr, 36 op/s 2024-09-21T05:46:54.096 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:53 smithi181 ceph-mon[34819]: pgmap v321: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 84 KiB/s wr, 36 op/s 2024-09-21T05:46:54.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:53 smithi129 ceph-mon[34885]: pgmap v321: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 84 KiB/s wr, 36 op/s 2024-09-21T05:46:56.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:55 smithi028 ceph-mon[31227]: pgmap v322: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-09-21T05:46:56.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:55 smithi129 ceph-mon[34885]: pgmap v322: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-09-21T05:46:56.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:55 smithi181 ceph-mon[34819]: pgmap v322: 97 pgs: 97 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-09-21T05:46:58.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:57 smithi129 ceph-mon[34885]: pgmap v323: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-21T05:46:58.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:57 smithi181 ceph-mon[34819]: pgmap v323: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-21T05:46:58.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:57 smithi028 ceph-mon[31227]: pgmap v323: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-21T05:47:00.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:46:59 smithi129 ceph-mon[34885]: pgmap v324: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-09-21T05:47:00.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:46:59 smithi028 ceph-mon[31227]: pgmap v324: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-09-21T05:47:00.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:46:59 smithi181 ceph-mon[34819]: pgmap v324: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 15 op/s 2024-09-21T05:47:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:47:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:47:00.117+0000 7f3f64503640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:47:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:47:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:47:00.117+0000 7f3f64503640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:47:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:47:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:47:00.117+0000 7f3f64503640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:47:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:47:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:47:00.117+0000 7f3f64503640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:47:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:47:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:47:00.117+0000 7f3f64503640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T05:47:02.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:01 smithi129 ceph-mon[34885]: pgmap v325: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-21T05:47:02.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:01 smithi129 ceph-mon[34885]: mgrmap e25: x(active, since 8m), standbys: z, y 2024-09-21T05:47:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:01 smithi181 ceph-mon[34819]: pgmap v325: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-21T05:47:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:01 smithi181 ceph-mon[34819]: mgrmap e25: x(active, since 8m), standbys: z, y 2024-09-21T05:47:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:01 smithi028 ceph-mon[31227]: pgmap v325: 97 pgs: 97 active+clean; 1.4 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-21T05:47:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:01 smithi028 ceph-mon[31227]: mgrmap e25: x(active, since 8m), standbys: z, y 2024-09-21T05:47:04.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:03 smithi028 ceph-mon[31227]: pgmap v326: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-21T05:47:04.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:03 smithi129 ceph-mon[34885]: pgmap v326: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-21T05:47:04.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:03 smithi181 ceph-mon[34819]: pgmap v326: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-21T05:47:06.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:05 smithi129 ceph-mon[34885]: pgmap v327: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:06.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 -- 172.21.15.28:0/460444236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 msgr2=0x7f6a88102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 --2- 172.21.15.28:0/460444236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a88102e20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009900 tx=0x7f6a8402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:06.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 -- 172.21.15.28:0/460444236 shutdown_connections 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 --2- 172.21.15.28:0/460444236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8810f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 --2- 172.21.15.28:0/460444236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 0x7f6a88108750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 --2- 172.21.15.28:0/460444236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a88102e20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 -- 172.21.15.28:0/460444236 >> 172.21.15.28:0/460444236 conn(0x7f6a880fe410 msgr2=0x7f6a88100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 -- 172.21.15.28:0/460444236 shutdown_connections 2024-09-21T05:47:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.222+0000 7f6a9000e640 1 -- 172.21.15.28:0/460444236 wait complete. 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.223+0000 7f6a9000e640 1 Processor -- start 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.223+0000 7f6a9000e640 1 -- start start 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 0x7f6a8819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88113ae0 con 0x7f6a88102a10 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88113960 con 0x7f6a88108c90 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a9000e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88113c60 con 0x7f6a88108340 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59392/0 (socket says 172.21.15.28:59392) 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1111536051 learned_addr learned my addr 172.21.15.28:0/1111536051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 0x7f6a8819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1111536051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 msgr2=0x7f6a8819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 0x7f6a8819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1111536051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 msgr2=0x7f6a8819fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8819fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1111536051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a881a0450 con 0x7f6a88102a10 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8819fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005ab0 tx=0x7f6a84003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.224+0000 7f6a7effd640 1 -- 172.21.15.28:0/1111536051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8403f070 con 0x7f6a88102a10 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.225+0000 7f6a7effd640 1 -- 172.21.15.28:0/1111536051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a84031d60 con 0x7f6a88102a10 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.225+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881a06e0 con 0x7f6a88102a10 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.225+0000 7f6a7effd640 1 -- 172.21.15.28:0/1111536051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84035030 con 0x7f6a88102a10 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.225+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a881a6ed0 con 0x7f6a88102a10 2024-09-21T05:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.227+0000 7f6a7effd640 1 -- 172.21.15.28:0/1111536051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f6a84042070 con 0x7f6a88102a10 2024-09-21T05:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.228+0000 7f6a7effd640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 0x7f6a740b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.228+0000 7f6a97056740 1 Processor -- start 2024-09-21T05:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.228+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 0x7f6a740b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.228+0000 7f6a7effd640 1 -- 172.21.15.28:0/1111536051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6a84108000 con 0x7f6a88102a10 2024-09-21T05:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.228+0000 7f6a97056740 1 -- start start 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77321b0 0x5649a7ad79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 0x7f6a740b8b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a88106f60 tx=0x7f6a78009110 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ad7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649a7a4c6f0 con 0x5649a77114d0 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649a7a3bb20 con 0x5649a77092f0 2024-09-21T05:47:06.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a97056740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649a7a3f930 con 0x5649a77321b0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59396/0 (socket says 172.21.15.28:59396) 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/34770777 learned_addr learned my addr 172.21.15.28:0/34770777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ad7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77321b0 0x5649a7ad79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/34770777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77321b0 msgr2=0x5649a7ad79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77321b0 0x5649a7ad79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/34770777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 msgr2=0x5649a7ad7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ad7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/34770777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a7a01530 con 0x5649a77114d0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.229+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ad7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.230+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009fb0 tx=0x7f6a80010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a13f7a381b2486 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.230+0000 7f6a73fff640 1 -- 172.21.15.28:0/34770777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80016070 con 0x5649a77114d0 2024-09-21T05:47:06.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.230+0000 7f6a73fff640 1 -- 172.21.15.28:0/34770777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a80012bc0 con 0x5649a77114d0 2024-09-21T05:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.230+0000 7f6a73fff640 1 -- 172.21.15.28:0/34770777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80011bd0 con 0x5649a77114d0 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 -- 172.21.15.28:0/34770777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 msgr2=0x5649a7ad8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009fb0 tx=0x7f6a80010800 comp rx=0 tx=0).stop 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 -- 172.21.15.28:0/34770777 shutdown_connections 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7ad8430 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ad7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 --2- 172.21.15.28:0/34770777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77321b0 0x5649a7ad79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 -- 172.21.15.28:0/34770777 >> 172.21.15.28:0/34770777 conn(0x5649a7726300 msgr2=0x5649a7ad58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 -- 172.21.15.28:0/34770777 shutdown_connections 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.231+0000 7f6a97056740 1 -- 172.21.15.28:0/34770777 wait complete. 2024-09-21T05:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 -- start start 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 Processor -- start 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ae3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.232+0000 7f6a97056740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649a7973ab0 con 0x5649a77321b0 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a97056740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649a79e1730 con 0x5649a77092f0 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a97056740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649a7ae6130 con 0x5649a77114d0 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a97056740 10 client.0 _ll_register_callbacks cb 0x5649a7a356b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59400/0 (socket says 172.21.15.28:59400) 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/767958951 learned_addr learned my addr 172.21.15.28:0/767958951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ae3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 msgr2=0x5649a7ae35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/767958951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 msgr2=0x5649a7ae3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ae3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/767958951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a779d790 con 0x5649a77321b0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009cf0 tx=0x7f6a80009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bbb7daf95b9ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80029060 con 0x5649a77321b0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6a80004c50 con 0x5649a77321b0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.233+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a80012bc0 con 0x5649a77321b0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6a80016070 con 0x5649a77321b0 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 10 client.15621 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649a7816620 con 0x5649a77321b0 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a80012bc0 con 0x5649a77321b0 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6a8000f040 con 0x5649a77321b0 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 10 client.15621 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 20 client.15621 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a76f0d80 0x5649a7af50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 4 client.15621 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.234+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649a7a4e640 con 0x5649a76f0d80 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.235+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a76f0d80 0x5649a7af50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.235+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a76f0d80 0x5649a7af50e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b6c204521c6fd80 server_cookie=6fd5a06424d543dd in_seq=0 out_seq=0 2024-09-21T05:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.235+0000 7f6a717fa640 10 client.15621 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a717fa640 1 -- 172.21.15.28:0/767958951 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5649a7a4e640 con 0x5649a76f0d80 2024-09-21T05:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a717fa640 10 client.15621 handle_command_reply: tid=0 2024-09-21T05:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 client.15621 shutdown 2024-09-21T05:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 msgr2=0x5649a7aef450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009cf0 tx=0x7f6a80009280 comp rx=0 tx=0).stop 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 shutdown_connections 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a76f0d80 0x5649a7af50e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77321b0 0x5649a7aef450 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a717fa640 0 client.15621 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 --2- 172.21.15.28:0/767958951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7ae3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 >> 172.21.15.28:0/767958951 conn(0x5649a7726300 msgr2=0x5649a7ad89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 shutdown_connections 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 1 -- 172.21.15.28:0/767958951 wait complete. 2024-09-21T05:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.236+0000 7f6a97056740 20 client.15621 trim_cache size 0 max 16384 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 Processor -- start 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 -- start start 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7aef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881af780 con 0x5649a76f0d80 2024-09-21T05:47:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881b12c0 con 0x5649a77092f0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a97056740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881afe50 con 0x5649a77114d0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a8ddaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a8ddaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59414/0 (socket says 172.21.15.28:59414) 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1231826709 learned_addr learned my addr 172.21.15.28:0/1231826709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.251+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7aef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1231826709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 msgr2=0x5649a7ae4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1231826709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 msgr2=0x5649a7aef3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7aef3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 -- 172.21.15.28:0/1231826709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a7add5b0 con 0x5649a76f0d80 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6a840cdb70 tx=0x7f6a840f1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3114113775d6b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a737fe640 1 -- 172.21.15.28:0/1231826709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84049040 con 0x5649a76f0d80 2024-09-21T05:47:06.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a737fe640 1 -- 172.21.15.28:0/1231826709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a840d7890 con 0x5649a76f0d80 2024-09-21T05:47:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.252+0000 7f6a737fe640 1 -- 172.21.15.28:0/1231826709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8403b6d0 con 0x5649a76f0d80 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- 172.21.15.28:0/1231826709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 msgr2=0x5649a7ae3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6a840cdb70 tx=0x7f6a840f1040 comp rx=0 tx=0).stop 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- 172.21.15.28:0/1231826709 shutdown_connections 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a77114d0 0x5649a7ae4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7aef3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 --2- 172.21.15.28:0/1231826709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a76f0d80 0x5649a7ae3560 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- 172.21.15.28:0/1231826709 >> 172.21.15.28:0/1231826709 conn(0x5649a7726300 msgr2=0x5649a7ae45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- 172.21.15.28:0/1231826709 shutdown_connections 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- 172.21.15.28:0/1231826709 wait complete. 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.253+0000 7f6a97056740 1 -- start start 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 Processor -- start 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a76f0d80 0x5649a7af5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7af6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881b0bf0 con 0x5649a77114d0 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881b0520 con 0x5649a77092f0 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881ad760 con 0x5649a76f0d80 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a97056740 10 client.0 _ll_register_callbacks cb 0x7f6a780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a8e5ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59416/0 (socket says 172.21.15.28:59416) 2024-09-21T05:47:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/3787984912 learned_addr learned my addr 172.21.15.28:0/3787984912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a76f0d80 0x5649a7af5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.254+0000 7f6a8d5a9640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7af6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a76f0d80 msgr2=0x5649a7af5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a76f0d80 0x5649a7af5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/3787984912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 msgr2=0x5649a7af6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7af6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 -- 172.21.15.28:0/3787984912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a7b56eb0 con 0x5649a77114d0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a8e5ab640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006010 tx=0x7f6a80004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b33ce09ff6cf3a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8003a060 con 0x5649a77114d0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6a80012b60 con 0x5649a77114d0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a840d7890 con 0x5649a77114d0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6a80029030 con 0x5649a77114d0 2024-09-21T05:47:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a97056740 10 client.15627 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.255+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649a7b5f340 con 0x5649a77114d0 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a840d7890 con 0x5649a77114d0 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6a800116e0 con 0x5649a77114d0 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 10 client.15627 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 20 client.15627 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a77321b0 0x5649a7b61ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 4 client.15627 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649a7a4a0f0 con 0x5649a77321b0 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.256+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a77321b0 0x5649a7b61ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.257+0000 7f6a717fa640 10 client.15627 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.257+0000 7f6a8ddaa640 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a77321b0 0x5649a7b61ad0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=179138331fbcb191 server_cookie=e95826d90f600737 in_seq=0 out_seq=0 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.264+0000 7f6a717fa640 1 -- 172.21.15.28:0/3787984912 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649a7a4a0f0 con 0x5649a77321b0 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.264+0000 7f6a717fa640 10 client.15627 handle_command_reply: tid=0 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.264+0000 7f6a97056740 1 client.15627 shutdown 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.264+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 msgr2=0x5649a7b56810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006010 tx=0x7f6a80004fa0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 shutdown_connections 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649a77321b0 0x5649a7b61ad0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a717fa640 0 client.15627 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649a77114d0 0x5649a7b56810 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649a77092f0 0x5649a7af6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 --2- 172.21.15.28:0/3787984912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649a76f0d80 0x5649a7af5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 >> 172.21.15.28:0/3787984912 conn(0x5649a7726300 msgr2=0x5649a7ad5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 shutdown_connections 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 1 -- 172.21.15.28:0/3787984912 wait complete. 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a97056740 20 client.15627 trim_cache size 0 max 16384 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:47:06.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:47:06.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 msgr2=0x7f6a740b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 0x7f6a740b8b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a88106f60 tx=0x7f6a78009110 comp rx=0 tx=0).stop 2024-09-21T05:47:06.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 msgr2=0x7f6a8819a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:06.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.265+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6a84005ab0 tx=0x7f6a84003cd0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 shutdown_connections 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a88108c90 0x7f6a8819fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a88108340 0x7f6a8819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6a740b6680 0x7f6a740b8b90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 --2- 172.21.15.28:0/1111536051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a88102a10 0x7f6a8819a520 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 >> 172.21.15.28:0/1111536051 conn(0x7f6a880fe410 msgr2=0x7f6a88100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 shutdown_connections 2024-09-21T05:47:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:06.266+0000 7f6a9000e640 1 -- 172.21.15.28:0/1111536051 wait complete. 2024-09-21T05:47:06.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:05 smithi181 ceph-mon[34819]: pgmap v327: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:06.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:05 smithi028 ceph-mon[31227]: pgmap v327: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:08.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:07 smithi129 ceph-mon[34885]: pgmap v328: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-21T05:47:08.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:07 smithi181 ceph-mon[34819]: pgmap v328: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-21T05:47:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:07 smithi028 ceph-mon[31227]: pgmap v328: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-21T05:47:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:09 smithi129 ceph-mon[34885]: pgmap v329: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:10.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:09 smithi181 ceph-mon[34819]: pgmap v329: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:09 smithi028 ceph-mon[31227]: pgmap v329: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-09-21T05:47:11.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:47:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:12.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:11 smithi028 ceph-mon[31227]: pgmap v330: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-09-21T05:47:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:11 smithi129 ceph-mon[34885]: pgmap v330: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-09-21T05:47:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:11 smithi181 ceph-mon[34819]: pgmap v330: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 -- 172.21.15.28:0/3266545920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f870810de10 msgr2=0x7f8708110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 --2- 172.21.15.28:0/3266545920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f870810de10 0x7f8708110300 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ca0 tx=0x7f870002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 -- 172.21.15.28:0/3266545920 shutdown_connections 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 --2- 172.21.15.28:0/3266545920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f870810de10 0x7f8708110300 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 --2- 172.21.15.28:0/3266545920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8708105300 0x7f870810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 --2- 172.21.15.28:0/3266545920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87081049b0 0x7f8708104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 -- 172.21.15.28:0/3266545920 >> 172.21.15.28:0/3266545920 conn(0x7f8708079730 msgr2=0x7f8708079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 -- 172.21.15.28:0/3266545920 shutdown_connections 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.718+0000 7f870f809640 1 -- 172.21.15.28:0/3266545920 wait complete. 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 Processor -- start 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 -- start start 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 0x7f870819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 0x7f870819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8708113a70 con 0x7f8708105300 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87081138f0 con 0x7f87081049b0 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870f809640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8708113bf0 con 0x7f870810de10 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.719+0000 7f870cda4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59446/0 (socket says 172.21.15.28:59446) 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 -- 172.21.15.28:0/2747076916 learned_addr learned my addr 172.21.15.28:0/2747076916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870d5a5640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 0x7f870819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870dda6640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 0x7f870819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 -- 172.21.15.28:0/2747076916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 msgr2=0x7f870819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 0x7f870819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 -- 172.21.15.28:0/2747076916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 msgr2=0x7f870819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 0x7f870819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 -- 172.21.15.28:0/2747076916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87081a0390 con 0x7f8708105300 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870dda6640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 0x7f870819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870d5a5640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 0x7f870819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870cda4640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f86f800d740 tx=0x7f86f800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f86fe7fc640 1 -- 172.21.15.28:0/2747076916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f8017070 con 0x7f8708105300 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f86fe7fc640 1 -- 172.21.15.28:0/2747076916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86f800de50 con 0x7f8708105300 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87081a0680 con 0x7f8708105300 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.720+0000 7f86fe7fc640 1 -- 172.21.15.28:0/2747076916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f800f620 con 0x7f8708105300 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.721+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87081a6de0 con 0x7f8708105300 2024-09-21T05:47:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.723+0000 7f86fe7fc640 1 -- 172.21.15.28:0/2747076916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f86f801a070 con 0x7f8708105300 2024-09-21T05:47:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.724+0000 7f8715d7d740 1 Processor -- start 2024-09-21T05:47:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.724+0000 7f8715d7d740 1 -- start start 2024-09-21T05:47:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.724+0000 7f86fe7fc640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 0x7f86f40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.724+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.724+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb851a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8152390 0x556fb851acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f8715d7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556fb8490e50 con 0x556fb814a1b0 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f8715d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556fb8480280 con 0x556fb8152390 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f8715d7d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556fb8484090 con 0x556fb8173070 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870dda6640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 0x7f86f40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f86fe7fc640 1 -- 172.21.15.28:0/2747076916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f86f80e7110 con 0x7f8708105300 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41136/0 (socket says 172.21.15.28:41136) 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 -- 172.21.15.28:0/3505990173 learned_addr learned my addr 172.21.15.28:0/3505990173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870cda4640 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb851a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 -- 172.21.15.28:0/3505990173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8152390 msgr2=0x556fb851acf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8152390 0x556fb851acf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 -- 172.21.15.28:0/3505990173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 msgr2=0x556fb851a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb851a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870d5a5640 1 -- 172.21.15.28:0/3505990173 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fb81de650 con 0x556fb8173070 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.725+0000 7f870dda6640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 0x7f86f40b8b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8700006010 tx=0x7f87000093f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.726+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f870400bd40 tx=0x7f870400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b51dae460ac4f934 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.726+0000 7f86f37fe640 1 -- 172.21.15.28:0/3505990173 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704011640 con 0x556fb8173070 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.726+0000 7f86f37fe640 1 -- 172.21.15.28:0/3505990173 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f870400be30 con 0x556fb8173070 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.726+0000 7f86f37fe640 1 -- 172.21.15.28:0/3505990173 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704010ab0 con 0x556fb8173070 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- 172.21.15.28:0/3505990173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 msgr2=0x556fb851a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f870400bd40 tx=0x7f870400f730 comp rx=0 tx=0).stop 2024-09-21T05:47:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- 172.21.15.28:0/3505990173 shutdown_connections 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8152390 0x556fb851acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb851a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3505990173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8173070 0x556fb851a270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- 172.21.15.28:0/3505990173 >> 172.21.15.28:0/3505990173 conn(0x556fb8167460 msgr2=0x556fb8518160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- 172.21.15.28:0/3505990173 shutdown_connections 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- 172.21.15.28:0/3505990173 wait complete. 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 -- start start 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.727+0000 7f8715d7d740 1 Processor -- start 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb8525930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb8525e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556fb82e1700 con 0x556fb8173070 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556fb841b110 con 0x556fb814a1b0 2024-09-21T05:47:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556fb8528a50 con 0x556fb8152390 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f8715d7d740 10 client.0 _ll_register_callbacks cb 0x556fb8434990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f870dda6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb8525930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f870dda6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59456/0 (socket says 172.21.15.28:59456) 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f870dda6640 1 -- 172.21.15.28:0/1713453073 learned_addr learned my addr 172.21.15.28:0/1713453073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.728+0000 7f870cda4640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb8525e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 -- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 msgr2=0x556fb8525e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb8525e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 -- 172.21.15.28:0/1713453073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 msgr2=0x556fb8525930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb8525930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 -- 172.21.15.28:0/1713453073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fb8448c40 con 0x556fb8173070 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f870dda6640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8700009a60 tx=0x7f870002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a682cbf510505b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f86f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700004230 con 0x556fb8173070 2024-09-21T05:47:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f87000043d0 con 0x556fb8173070 2024-09-21T05:47:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.729+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f870400be30 con 0x556fb8173070 2024-09-21T05:47:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8700058050 con 0x556fb8173070 2024-09-21T05:47:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 10 client.15639 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fb8534760 con 0x556fb8173070 2024-09-21T05:47:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f870400be30 con 0x556fb8173070 2024-09-21T05:47:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f87000305c0 con 0x556fb8173070 2024-09-21T05:47:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 10 client.15639 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 20 client.15639 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb81319f0 0x556fb8537ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 4 client.15639 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556fb8492da0 con 0x556fb81319f0 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.730+0000 7f870d5a5640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb81319f0 0x556fb8537ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.731+0000 7f870d5a5640 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb81319f0 0x556fb8537ce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aff5ee37b613c33 server_cookie=ef34e4dd366b0662 in_seq=0 out_seq=0 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.731+0000 7f86f0ff9640 10 client.15639 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.731+0000 7f86f0ff9640 1 -- 172.21.15.28:0/1713453073 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556fb8492da0 con 0x556fb81319f0 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.731+0000 7f86f0ff9640 10 client.15639 handle_command_reply: tid=0 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 client.15639 shutdown 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 msgr2=0x556fb8531d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8700009a60 tx=0x7f870002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 shutdown_connections 2024-09-21T05:47:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb81319f0 0x556fb8537ce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8531d40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb8525e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 --2- 172.21.15.28:0/1713453073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb8525930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 >> 172.21.15.28:0/1713453073 conn(0x556fb8167460 msgr2=0x556fb851d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f86f0ff9640 0 client.15639 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 shutdown_connections 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 1 -- 172.21.15.28:0/1713453073 wait complete. 2024-09-21T05:47:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.732+0000 7f8715d7d740 20 client.15639 trim_cache size 0 max 16384 2024-09-21T05:47:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 Processor -- start 2024-09-21T05:47:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 -- start start 2024-09-21T05:47:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb851b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 0x556fb8517710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87081af770 con 0x556fb814a1b0 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87081b12b0 con 0x556fb8173070 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f8715d7d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87081afe40 con 0x556fb8152390 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.764+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59458/0 (socket says 172.21.15.28:59458) 2024-09-21T05:47:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 -- 172.21.15.28:0/3820425272 learned_addr learned my addr 172.21.15.28:0/3820425272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870dda6640 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 0x556fb8517710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 -- 172.21.15.28:0/3820425272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 msgr2=0x556fb851b9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb851b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 -- 172.21.15.28:0/3820425272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 msgr2=0x556fb8517710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 0x556fb8517710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 -- 172.21.15.28:0/3820425272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fb8526bf0 con 0x556fb814a1b0 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870dda6640 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 0x556fb8517710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.765+0000 7f870d5a5640 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8704010660 tx=0x7f87040105f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc820ea44b559bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.766+0000 7f86f2ffd640 1 -- 172.21.15.28:0/3820425272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704005040 con 0x556fb814a1b0 2024-09-21T05:47:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.766+0000 7f86f2ffd640 1 -- 172.21.15.28:0/3820425272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8704010700 con 0x556fb814a1b0 2024-09-21T05:47:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.766+0000 7f86f2ffd640 1 -- 172.21.15.28:0/3820425272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87040048e0 con 0x556fb814a1b0 2024-09-21T05:47:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- 172.21.15.28:0/3820425272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 msgr2=0x556fb8524820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8704010660 tx=0x7f87040105f0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- 172.21.15.28:0/3820425272 shutdown_connections 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb8173070 0x556fb8517710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb851b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 --2- 172.21.15.28:0/3820425272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb814a1b0 0x556fb8524820 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- 172.21.15.28:0/3820425272 >> 172.21.15.28:0/3820425272 conn(0x556fb8167460 msgr2=0x556fb851b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- 172.21.15.28:0/3820425272 shutdown_connections 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- 172.21.15.28:0/3820425272 wait complete. 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.767+0000 7f8715d7d740 1 -- start start 2024-09-21T05:47:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 Processor -- start 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb852e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb85298a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87081b0be0 con 0x556fb8173070 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87081b0510 con 0x556fb814a1b0 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87081ad750 con 0x556fb8152390 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f8715d7d740 10 client.0 _ll_register_callbacks cb 0x7f8700009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.768+0000 7f870dda6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59472/0 (socket says 172.21.15.28:59472) 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870d5a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb852e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 -- 172.21.15.28:0/2129647778 learned_addr learned my addr 172.21.15.28:0/2129647778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870cda4640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb85298a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 -- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 msgr2=0x556fb85298a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb85298a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 -- 172.21.15.28:0/2129647778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 msgr2=0x556fb852e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb852e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 -- 172.21.15.28:0/2129647778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fb852a4d0 con 0x556fb8173070 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f870dda6640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8700009600 tx=0x7f8700030e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dcc31ea9ae6b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f86f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700060070 con 0x556fb8173070 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8700004230 con 0x556fb8173070 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.769+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8704010700 con 0x556fb8173070 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.770+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8700031e90 con 0x556fb8173070 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.770+0000 7f8715d7d740 10 client.15645 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.770+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fb85a2590 con 0x556fb8173070 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.770+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8704010700 con 0x556fb8173070 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.770+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f870002fab0 con 0x556fb8173070 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 10 client.15645 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 10 client.15645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 20 client.15645 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb8106da0 0x556fb85a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 4 client.15645 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556fb848e850 con 0x556fb8106da0 2024-09-21T05:47:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f870d5a5640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb8106da0 0x556fb85a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f870d5a5640 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb8106da0 0x556fb85a4cf0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d9779dab124bab9 server_cookie=10469e4cd76b7b9b in_seq=0 out_seq=0 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.771+0000 7f86f0ff9640 10 client.15645 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f86f0ff9640 1 -- 172.21.15.28:0/2129647778 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+374 (crc 0 0 0) 0x556fb848e850 con 0x556fb8106da0 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f86f0ff9640 10 client.15645 handle_command_reply: tid=0 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 client.15645 shutdown 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 msgr2=0x556fb8529de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8700009600 tx=0x7f8700030e80 comp rx=0 tx=0).stop 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 shutdown_connections 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556fb8106da0 0x556fb85a4cf0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556fb8173070 0x556fb8529de0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556fb8152390 0x556fb85298a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 --2- 172.21.15.28:0/2129647778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556fb814a1b0 0x556fb852e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 >> 172.21.15.28:0/2129647778 conn(0x556fb8167460 msgr2=0x556fb8527780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f86f0ff9640 0 client.15645 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 shutdown_connections 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 1 -- 172.21.15.28:0/2129647778 wait complete. 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.772+0000 7f8715d7d740 20 client.15645 trim_cache size 0 max 16384 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:12.777 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 msgr2=0x7f86f40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 0x7f86f40b8b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8700006010 tx=0x7f87000093f0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 msgr2=0x7f870819a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f86f800d740 tx=0x7f86f800dc00 comp rx=0 tx=0).stop 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 shutdown_connections 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f870810de10 0x7f870819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f86f40b6680 0x7f86f40b8b90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8708105300 0x7f870819a9c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 --2- 172.21.15.28:0/2747076916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87081049b0 0x7f870819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 >> 172.21.15.28:0/2747076916 conn(0x7f8708079730 msgr2=0x7f87080ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.773+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 shutdown_connections 2024-09-21T05:47:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:12.774+0000 7f870f809640 1 -- 172.21.15.28:0/2747076916 wait complete. 2024-09-21T05:47:13.688 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:47:13.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:47:13.970 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:14.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:13 smithi028 ceph-mon[31227]: pgmap v331: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-09-21T05:47:14.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:13 smithi129 ceph-mon[34885]: pgmap v331: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-09-21T05:47:14.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:13 smithi181 ceph-mon[34819]: pgmap v331: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 -- 172.21.15.28:0/3839286627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 msgr2=0x7f5e1c10fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/3839286627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c10fd80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009900 tx=0x7f5e1002a480 comp rx=0 tx=0).stop 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 -- 172.21.15.28:0/3839286627 shutdown_connections 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/3839286627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 0x7f5e1c1126e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/3839286627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c10fd80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/3839286627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 0x7f5e1c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 -- 172.21.15.28:0/3839286627 >> 172.21.15.28:0/3839286627 conn(0x7f5e1c079750 msgr2=0x7f5e1c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 -- 172.21.15.28:0/3839286627 shutdown_connections 2024-09-21T05:47:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.570+0000 7f5e20c0f640 1 -- 172.21.15.28:0/3839286627 wait complete. 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.571+0000 7f5e20c0f640 1 Processor -- start 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.571+0000 7f5e20c0f640 1 -- start start 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 0x7f5e1c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 0x7f5e1c1a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1c115c90 con 0x7f5e1c103060 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1c115b10 con 0x7f5e1c1102c0 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e20c0f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1c115e10 con 0x7f5e1c102710 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e19d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e19d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59488/0 (socket says 172.21.15.28:59488) 2024-09-21T05:47:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e19d9b640 1 -- 172.21.15.28:0/2795967162 learned_addr learned my addr 172.21.15.28:0/2795967162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e1ad9d640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 0x7f5e1c1a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.572+0000 7f5e1a59c640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 0x7f5e1c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 -- 172.21.15.28:0/2795967162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 msgr2=0x7f5e1c19c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 0x7f5e1c19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 -- 172.21.15.28:0/2795967162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 msgr2=0x7f5e1c1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 0x7f5e1c1a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 -- 172.21.15.28:0/2795967162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1c1a2630 con 0x7f5e1c103060 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.573+0000 7f5e19d9b640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005940 tx=0x7f5e100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.574+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1003f030 con 0x7f5e1c103060 2024-09-21T05:47:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.574+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e10003cb0 con 0x7f5e1c103060 2024-09-21T05:47:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.574+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1c1a28c0 con 0x7f5e1c103060 2024-09-21T05:47:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.574+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10035070 con 0x7f5e1c103060 2024-09-21T05:47:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.574+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e1c1a9080 con 0x7f5e1c103060 2024-09-21T05:47:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.575+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e1c06c1e0 con 0x7f5e1c103060 2024-09-21T05:47:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.578+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5e10042030 con 0x7f5e1c103060 2024-09-21T05:47:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.579+0000 7f5e0b7fe640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 0x7f5e040b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.579+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5e101081f0 con 0x7f5e1c103060 2024-09-21T05:47:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.579+0000 7f5e1a59c640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 0x7f5e040b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.580+0000 7f5e1a59c640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 0x7f5e040b8c90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c005fd0 tx=0x7f5e0c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.583+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5e100ce700 con 0x7f5e1c103060 2024-09-21T05:47:14.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.911+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e1c102b20 con 0x7f5e1c103060 2024-09-21T05:47:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.912+0000 7f5e0b7fe640 1 -- 172.21.15.28:0/2795967162 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7f5e100e7ef0 con 0x7f5e1c103060 2024-09-21T05:47:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:47:14.915 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:47:14.915 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 msgr2=0x7f5e040b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 0x7f5e040b8c90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c005fd0 tx=0x7f5e0c005850 comp rx=0 tx=0).stop 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 msgr2=0x7f5e1c19cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005940 tx=0x7f5e100042f0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 shutdown_connections 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c1102c0 0x7f5e1c1a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5e040b6780 0x7f5e040b8c90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e1c103060 0x7f5e1c19cc60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 --2- 172.21.15.28:0/2795967162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c102710 0x7f5e1c19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 >> 172.21.15.28:0/2795967162 conn(0x7f5e1c079750 msgr2=0x7f5e1c108dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 shutdown_connections 2024-09-21T05:47:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:14.916+0000 7f5e20c0f640 1 -- 172.21.15.28:0/2795967162 wait complete. 2024-09-21T05:47:15.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:47:15.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:16.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:15 smithi028 ceph-mon[31227]: pgmap v332: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T05:47:16.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:15 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2795967162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:16.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:15 smithi129 ceph-mon[34885]: pgmap v332: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T05:47:16.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:15 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2795967162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:16.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:15 smithi181 ceph-mon[34819]: pgmap v332: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-09-21T05:47:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:15 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2795967162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.372+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2150726600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b01049d0 msgr2=0x7fc6b0104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.372+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2150726600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b01049d0 0x7fc6b0104de0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009900 tx=0x7fc6ac02a410 comp rx=0 tx=0).stop 2024-09-21T05:47:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2150726600 shutdown_connections 2024-09-21T05:47:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2150726600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b010de30 0x7fc6b0110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2150726600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b0105320 0x7fc6b010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2150726600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b01049d0 0x7fc6b0104de0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2150726600 >> 172.21.15.28:0/2150726600 conn(0x7fc6b0079740 msgr2=0x7fc6b0079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2150726600 shutdown_connections 2024-09-21T05:47:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.373+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2150726600 wait complete. 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 Processor -- start 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 -- start start 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 0x7fc6b019a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b010de30 0x7fc6b019fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0113ae0 con 0x7fc6b0105320 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0113960 con 0x7fc6b01049d0 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.374+0000 7fc6b7d54640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0113c60 con 0x7fc6b010de30 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 0x7fc6b019a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44700/0 (socket says 172.21.15.28:44700) 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 -- 172.21.15.28:0/2699797322 learned_addr learned my addr 172.21.15.28:0/2699797322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b010de30 0x7fc6b019fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 -- 172.21.15.28:0/2699797322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b010de30 msgr2=0x7fc6b019fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b010de30 0x7fc6b019fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 -- 172.21.15.28:0/2699797322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 msgr2=0x7fc6b019a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 0x7fc6b019a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 -- 172.21.15.28:0/2699797322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b01a0460 con 0x7fc6b0105320 2024-09-21T05:47:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 0x7fc6b019a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.375+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000e9f0 tx=0x7fc6a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.376+0000 7fc6a6ffd640 1 -- 172.21.15.28:0/2699797322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a000ccc0 con 0x7fc6b0105320 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.376+0000 7fc6a6ffd640 1 -- 172.21.15.28:0/2699797322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a00108e0 con 0x7fc6b0105320 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.376+0000 7fc6a6ffd640 1 -- 172.21.15.28:0/2699797322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018860 con 0x7fc6b0105320 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.376+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b01a0750 con 0x7fc6b0105320 2024-09-21T05:47:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.376+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b01a6eb0 con 0x7fc6b0105320 2024-09-21T05:47:16.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.379+0000 7fc6a6ffd640 1 -- 172.21.15.28:0/2699797322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc6a0010430 con 0x7fc6b0105320 2024-09-21T05:47:16.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6be2c8740 1 Processor -- start 2024-09-21T05:47:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6be2c8740 1 -- start start 2024-09-21T05:47:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6a6ffd640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 0x7fc69c0b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 0x7fc69c0b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.380+0000 7fc6a6ffd640 1 -- 172.21.15.28:0/2699797322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc6a00e7930 con 0x7fc6b0105320 2024-09-21T05:47:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 0x5560aabb1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44710/0 (socket says 172.21.15.28:44710) 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40068/0 (socket says 172.21.15.28:40068) 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b52ef640 1 -- 172.21.15.28:0/3623173950 learned_addr learned my addr 172.21.15.28:0/3623173950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 0x5560aabb1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5560aab26ea0 con 0x5560aa7e01b0 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5560aab162d0 con 0x5560aa7e8390 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5560aab1a0e0 con 0x5560aa809070 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.381+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 0x7fc69c0b8ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8005fd0 tx=0x7fc6a8009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 -- 172.21.15.28:0/3623173950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 msgr2=0x5560aabb06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 -- 172.21.15.28:0/3623173950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 msgr2=0x5560aabb1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 0x5560aabb1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 -- 172.21.15.28:0/3623173950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560aa874650 con 0x5560aa7e01b0 2024-09-21T05:47:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 0x5560aabb1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00a9d70 tx=0x7fc6a000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be43ab93a0947d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc69bfff640 1 -- 172.21.15.28:0/3623173950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00b2280 con 0x5560aa7e01b0 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc69bfff640 1 -- 172.21.15.28:0/3623173950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a00b71c0 con 0x5560aa7e01b0 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.382+0000 7fc69bfff640 1 -- 172.21.15.28:0/3623173950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00ac740 con 0x5560aa7e01b0 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.383+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 msgr2=0x5560aabb0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.383+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00a9d70 tx=0x7fc6a000cef0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 shutdown_connections 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7e8390 0x5560aabb1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb0be0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3623173950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa809070 0x5560aabb06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 >> 172.21.15.28:0/3623173950 conn(0x5560aa7fd460 msgr2=0x5560aabae590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 shutdown_connections 2024-09-21T05:47:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3623173950 wait complete. 2024-09-21T05:47:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.384+0000 7fc6be2c8740 1 -- start start 2024-09-21T05:47:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.385+0000 7fc6be2c8740 1 Processor -- start 2024-09-21T05:47:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabbc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 0x5560aabc8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44718/0 (socket says 172.21.15.28:44718) 2024-09-21T05:47:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5560aa977700 con 0x5560aa7e01b0 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5560aaab1160 con 0x5560aa809070 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5560aabbee20 con 0x5560aa7e8390 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b5af0640 1 -- 172.21.15.28:0/1222270291 learned_addr learned my addr 172.21.15.28:0/1222270291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabbc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 0x5560aabc8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 -- 172.21.15.28:0/1222270291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 msgr2=0x5560aabbc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabbc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 -- 172.21.15.28:0/1222270291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 msgr2=0x5560aabbbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b62f1640 1 -- 172.21.15.28:0/1222270291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560aaadec90 con 0x5560aa809070 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.386+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabbc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6be2c8740 10 client.0 _ll_register_callbacks cb 0x5560aaaca9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6a00b71c0 con 0x5560aa809070 2024-09-21T05:47:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 0x5560aabc8110 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009d60 tx=0x7fc6a8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b257c07d74e0a0e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6997fa640 10 client.25292 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6997fa640 1 -- 172.21.15.28:0/1222270291 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a801dc80 con 0x5560aa809070 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6997fa640 1 -- 172.21.15.28:0/1222270291 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc6a8009040 con 0x5560aa809070 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6997fa640 1 -- 172.21.15.28:0/1222270291 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc6a801de20 con 0x5560aa809070 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6be2c8740 10 client.25292 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.387+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5560aabcaac0 con 0x5560aa809070 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6997fa640 1 -- 172.21.15.28:0/1222270291 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc6a8004b20 con 0x5560aa809070 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 10 client.25292 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 10 client.25292 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 20 client.25292 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa7c79f0 0x5560aabcdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 4 client.25292 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560aab28df0 con 0x5560aa7c79f0 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.388+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa7c79f0 0x5560aabcdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.389+0000 7fc6997fa640 10 client.25292 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.389+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa7c79f0 0x5560aabcdfb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=49bc8042c6206cf4 server_cookie=18795a11ac2b99f6 in_seq=0 out_seq=0 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6997fa640 1 -- 172.21.15.28:0/1222270291 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5560aab28df0 con 0x5560aa7c79f0 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6997fa640 10 client.25292 handle_command_reply: tid=0 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 client.25292 shutdown 2024-09-21T05:47:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 msgr2=0x5560aabc8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 0x5560aabc8110 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009d60 tx=0x7fc6a8004ab0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 shutdown_connections 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa7c79f0 0x5560aabcdfb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa809070 0x5560aabc8110 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabbc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6997fa640 0 client.25292 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/1222270291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabbbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 >> 172.21.15.28:0/1222270291 conn(0x5560aa7fd460 msgr2=0x5560aabb3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 shutdown_connections 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 1 -- 172.21.15.28:0/1222270291 wait complete. 2024-09-21T05:47:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.390+0000 7fc6be2c8740 20 client.25292 trim_cache size 0 max 16384 2024-09-21T05:47:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.403+0000 7fc6be2c8740 1 Processor -- start 2024-09-21T05:47:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.403+0000 7fc6be2c8740 1 -- start start 2024-09-21T05:47:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabb9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b1970 con 0x5560aa7e01b0 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b34b0 con 0x5560aa7c79f0 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6be2c8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b2040 con 0x5560aa7e8390 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6b62f1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabb9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60592/0 (socket says 172.21.15.28:60592) 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.404+0000 7fc6b5af0640 1 -- 172.21.15.28:0/4250655673 learned_addr learned my addr 172.21.15.28:0/4250655673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 -- 172.21.15.28:0/4250655673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 msgr2=0x5560aabb9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabb9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 -- 172.21.15.28:0/4250655673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 msgr2=0x5560aabb9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 -- 172.21.15.28:0/4250655673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560aabce510 con 0x5560aa7c79f0 2024-09-21T05:47:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006010 tx=0x7fc6ac000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703eb96e0b069f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc69b7fe640 1 -- 172.21.15.28:0/4250655673 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac035040 con 0x5560aa7c79f0 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc69b7fe640 1 -- 172.21.15.28:0/4250655673 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6ac004b70 con 0x5560aa7c79f0 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.405+0000 7fc69b7fe640 1 -- 172.21.15.28:0/4250655673 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac003bc0 con 0x5560aa7c79f0 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.406+0000 7fc6be2c8740 1 -- 172.21.15.28:0/4250655673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 msgr2=0x5560aabb1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.406+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006010 tx=0x7fc6ac000bc0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.406+0000 7fc6be2c8740 1 -- 172.21.15.28:0/4250655673 shutdown_connections 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.406+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e8390 0x5560aabb9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.406+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e01b0 0x5560aabb9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.407+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/4250655673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabb1de0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.407+0000 7fc6be2c8740 1 -- 172.21.15.28:0/4250655673 >> 172.21.15.28:0/4250655673 conn(0x5560aa7fd460 msgr2=0x5560aabb16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.407+0000 7fc6be2c8740 1 -- 172.21.15.28:0/4250655673 shutdown_connections 2024-09-21T05:47:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.407+0000 7fc6be2c8740 1 -- 172.21.15.28:0/4250655673 wait complete. 2024-09-21T05:47:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.407+0000 7fc6be2c8740 1 -- start start 2024-09-21T05:47:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.408+0000 7fc6be2c8740 1 Processor -- start 2024-09-21T05:47:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.408+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.408+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 0x5560aabc1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6be2c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 0x5560aac30a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b5af0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60600/0 (socket says 172.21.15.28:60600) 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 0x5560aabc1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b52ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 0x5560aabc1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40106/0 (socket says 172.21.15.28:40106) 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b62f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 0x5560aac30a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6b5af0640 1 -- 172.21.15.28:0/3372191722 learned_addr learned my addr 172.21.15.28:0/3372191722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b2de0 con 0x5560aa7e8390 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b2710 con 0x5560aa7c79f0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.409+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01af950 con 0x5560aa7e01b0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6be2c8740 10 client.0 _ll_register_callbacks cb 0x7fc6a800f4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 -- 172.21.15.28:0/3372191722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 msgr2=0x5560aabc1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 0x5560aabc1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 -- 172.21.15.28:0/3372191722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 msgr2=0x5560aac30a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 0x5560aac30a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560aac310e0 con 0x5560aa7c79f0 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b62f1640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 0x5560aac30a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6b5af0640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009900 tx=0x7fc6ac005360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b966fb154a46b59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6997fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.410+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac035040 con 0x5560aa7c79f0 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc6ac004d20 con 0x5560aa7c79f0 2024-09-21T05:47:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6ac004b70 con 0x5560aa7c79f0 2024-09-21T05:47:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc6ac003720 con 0x5560aa7c79f0 2024-09-21T05:47:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6be2c8740 10 client.25298 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560aac383c0 con 0x5560aa7c79f0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.411+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6ac004b70 con 0x5560aa7c79f0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc6ac0314a0 con 0x5560aa7c79f0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 10 client.25298 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 10 client.25298 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 20 client.25298 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa809070 0x5560aac3aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 4 client.25298 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560aab248a0 con 0x5560aa809070 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.412+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa809070 0x5560aac3aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6997fa640 10 client.25298 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6b52ef640 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa809070 0x5560aac3aa20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=687275c2435a0375 server_cookie=a1977044b43c343e in_seq=0 out_seq=0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6997fa640 1 -- 172.21.15.28:0/3372191722 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560aab248a0 con 0x5560aa809070 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6997fa640 10 client.25298 handle_command_reply: tid=0 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 client.25298 shutdown 2024-09-21T05:47:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 msgr2=0x5560aabc1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009900 tx=0x7fc6ac005360 comp rx=0 tx=0).stop 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 shutdown_connections 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5560aa809070 0x5560aac3aa20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560aa7e8390 0x5560aac30a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5560aa7e01b0 0x5560aabc1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 --2- 172.21.15.28:0/3372191722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5560aa7c79f0 0x5560aabc1a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6997fa640 0 client.25298 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 >> 172.21.15.28:0/3372191722 conn(0x5560aa7fd460 msgr2=0x5560aabb3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 shutdown_connections 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 1 -- 172.21.15.28:0/3372191722 wait complete. 2024-09-21T05:47:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.413+0000 7fc6be2c8740 20 client.25298 trim_cache size 0 max 16384 2024-09-21T05:47:16.419 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 msgr2=0x7fc69c0b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 0x7fc69c0b8ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8005fd0 tx=0x7fc6a8009500 comp rx=0 tx=0).stop 2024-09-21T05:47:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 msgr2=0x7fc6b019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000e9f0 tx=0x7fc6a000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 shutdown_connections 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b010de30 0x7fc6b019fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc69c0b67d0 0x7fc69c0b8ce0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b0105320 0x7fc6b019aa90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 --2- 172.21.15.28:0/2699797322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b01049d0 0x7fc6b019a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 >> 172.21.15.28:0/2699797322 conn(0x7fc6b0079740 msgr2=0x7fc6b01007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.414+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 shutdown_connections 2024-09-21T05:47:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:16.415+0000 7fc6b7d54640 1 -- 172.21.15.28:0/2699797322 wait complete. 2024-09-21T05:47:17.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:47:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 -- 172.21.15.28:0/2379897784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc94106c20 msgr2=0x7fbc94110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/2379897784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc94106c20 0x7fbc94110670 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009ca0 tx=0x7fbc8c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 -- 172.21.15.28:0/2379897784 shutdown_connections 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/2379897784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc94106c20 0x7fbc94110670 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/2379897784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc941062d0 0x7fbc941066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/2379897784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 0x7fbc94102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 -- 172.21.15.28:0/2379897784 >> 172.21.15.28:0/2379897784 conn(0x7fbc94079e30 msgr2=0x7fbc9407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 -- 172.21.15.28:0/2379897784 shutdown_connections 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.902+0000 7fbc9aae7640 1 -- 172.21.15.28:0/2379897784 wait complete. 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 Processor -- start 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 -- start start 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 0x7fbc941981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 0x7fbc9419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94111910 con 0x7fbc941062d0 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94111790 con 0x7fbc94106c20 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc9aae7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94111a90 con 0x7fbc94101ff0 2024-09-21T05:47:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44744/0 (socket says 172.21.15.28:44744) 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc93fff640 1 -- 172.21.15.28:0/1200243750 learned_addr learned my addr 172.21.15.28:0/1200243750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.903+0000 7fbc98883640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 0x7fbc941981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 -- 172.21.15.28:0/1200243750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 msgr2=0x7fbc941981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 0x7fbc941981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 -- 172.21.15.28:0/1200243750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 msgr2=0x7fbc9419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc99084640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 0x7fbc9419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 0x7fbc9419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 -- 172.21.15.28:0/1200243750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9419e150 con 0x7fbc941062d0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc99084640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 0x7fbc9419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc93fff640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbc80005fd0 tx=0x7fbc8000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc91ffb640 1 -- 172.21.15.28:0/1200243750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80017070 con 0x7fbc941062d0 2024-09-21T05:47:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc91ffb640 1 -- 172.21.15.28:0/1200243750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc80009c30 con 0x7fbc941062d0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9419e440 con 0x7fbc941062d0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.904+0000 7fbc91ffb640 1 -- 172.21.15.28:0/1200243750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80003d30 con 0x7fbc941062d0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.905+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc941a4ba0 con 0x7fbc941062d0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.908+0000 7fbc91ffb640 1 -- 172.21.15.28:0/1200243750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbc80003ed0 con 0x7fbc941062d0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.908+0000 7fbca105b740 1 Processor -- start 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.908+0000 7fbca105b740 1 -- start start 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfd8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfd9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ccf4eea0 con 0x55a5ccc31070 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a5ccf3e2d0 con 0x55a5ccc081b0 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbca105b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a5ccf420e0 con 0x55a5ccc10390 2024-09-21T05:47:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc91ffb640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 0x7fbc840b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44752/0 (socket says 172.21.15.28:44752) 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 -- 172.21.15.28:0/64511994 learned_addr learned my addr 172.21.15.28:0/64511994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 0x7fbc840b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc98883640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfd8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 -- 172.21.15.28:0/64511994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 msgr2=0x55a5ccfd9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc93fff640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfd9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.909+0000 7fbc99084640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfd9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc99084640 1 -- 172.21.15.28:0/64511994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 msgr2=0x55a5ccfd8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc99084640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfd8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc91ffb640 1 -- 172.21.15.28:0/1200243750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc800e7310 con 0x7fbc941062d0 2024-09-21T05:47:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc99084640 1 -- 172.21.15.28:0/64511994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ccc9c650 con 0x55a5ccc31070 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc98883640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfd8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc93fff640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfd9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc99084640 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0094e0 tx=0x7fbc8c031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d35090c750eaef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc99084640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 0x7fbc840b8c10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c046000 tx=0x7fbc8c048000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc7effd640 1 -- 172.21.15.28:0/64511994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c003c30 con 0x55a5ccc31070 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc7effd640 1 -- 172.21.15.28:0/64511994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c036070 con 0x55a5ccc31070 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.910+0000 7fbc7effd640 1 -- 172.21.15.28:0/64511994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c02f830 con 0x55a5ccc31070 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- 172.21.15.28:0/64511994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 msgr2=0x55a5ccfd86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0094e0 tx=0x7fbc8c031e60 comp rx=0 tx=0).stop 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- 172.21.15.28:0/64511994 shutdown_connections 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfd9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfd8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 --2- 172.21.15.28:0/64511994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccfd86a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- 172.21.15.28:0/64511994 >> 172.21.15.28:0/64511994 conn(0x55a5ccc25460 msgr2=0x55a5ccfd6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- 172.21.15.28:0/64511994 shutdown_connections 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- 172.21.15.28:0/64511994 wait complete. 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.912+0000 7fbca105b740 1 -- start start 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.913+0000 7fbca105b740 1 Processor -- start 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.913+0000 7fbca105b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.913+0000 7fbca105b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfe4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.913+0000 7fbca105b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccff00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.913+0000 7fbca105b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5ccd9f700 con 0x55a5ccc31070 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbca105b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a5cced9160 con 0x55a5ccc081b0 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbca105b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a5ccfe6dc0 con 0x55a5ccc10390 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60646/0 (socket says 172.21.15.28:60646) 2024-09-21T05:47:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 -- 172.21.15.28:0/3126222576 learned_addr learned my addr 172.21.15.28:0/3126222576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbca105b740 10 client.0 _ll_register_callbacks cb 0x55a5ccef29e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc93fff640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfe4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc99084640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccff00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 -- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 msgr2=0x55a5ccfe4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfe4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 -- 172.21.15.28:0/3126222576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 msgr2=0x55a5ccff00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccff00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc98883640 1 -- 172.21.15.28:0/3126222576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ccf06c90 con 0x55a5ccc081b0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.914+0000 7fbc99084640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccff00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbc98883640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800f8a0 tx=0x7fbc8800fd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7749fccf5bb043 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbc73fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880183f0 con 0x55a5ccc081b0 2024-09-21T05:47:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc88018590 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8c036070 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc88026050 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbca105b740 10 client.25304 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.915+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5ccff2a20 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.916+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c036070 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbc88021950 con 0x55a5ccc081b0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 10 client.25304 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 10 client.25304 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 20 client.25304 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccbef9f0 0x55a5ccff5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 4 client.25304 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5ccf50df0 con 0x55a5ccbef9f0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbc93fff640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccbef9f0 0x55a5ccff5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbc73fff640 10 client.25304 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.917+0000 7fbc93fff640 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccbef9f0 0x55a5ccff5ff0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dbe80bb8cf168089 server_cookie=89158fbd28cb9bca in_seq=0 out_seq=0 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.918+0000 7fbc73fff640 1 -- 172.21.15.28:0/3126222576 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a5ccf50df0 con 0x55a5ccbef9f0 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.918+0000 7fbc73fff640 10 client.25304 handle_command_reply: tid=0 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.918+0000 7fbca105b740 1 client.25304 shutdown 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 msgr2=0x55a5ccfe3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800f8a0 tx=0x7fbc8800fd60 comp rx=0 tx=0).stop 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 shutdown_connections 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccbef9f0 0x55a5ccff5ff0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc31070 0x55a5ccff00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccfe4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbc73fff640 0 client.25304 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 --2- 172.21.15.28:0/3126222576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc081b0 0x55a5ccfe3dc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 >> 172.21.15.28:0/3126222576 conn(0x55a5ccc25460 msgr2=0x55a5ccfdb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 shutdown_connections 2024-09-21T05:47:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 1 -- 172.21.15.28:0/3126222576 wait complete. 2024-09-21T05:47:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.919+0000 7fbca105b740 20 client.25304 trim_cache size 0 max 16384 2024-09-21T05:47:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.945+0000 7fbca105b740 1 Processor -- start 2024-09-21T05:47:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.945+0000 7fbca105b740 1 -- start start 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 0x55a5ccfd65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44772/0 (socket says 172.21.15.28:44772) 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc93fff640 1 -- 172.21.15.28:0/1127537984 learned_addr learned my addr 172.21.15.28:0/1127537984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc10390 0x55a5ccfe0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ad400 con 0x55a5ccc081b0 2024-09-21T05:47:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941aef40 con 0x55a5ccc10390 2024-09-21T05:47:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941adad0 con 0x55a5ccbef9f0 2024-09-21T05:47:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc98883640 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 0x55a5ccfd65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc98883640 1 -- 172.21.15.28:0/1127537984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc10390 msgr2=0x55a5ccfe0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.946+0000 7fbc98883640 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc10390 0x55a5ccfe0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc98883640 1 -- 172.21.15.28:0/1127537984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 msgr2=0x55a5ccff0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc98883640 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc98883640 1 -- 172.21.15.28:0/1127537984 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ccfe5080 con 0x55a5ccbef9f0 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc98883640 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 0x55a5ccfd65e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc88004400 tx=0x7fbc88005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c69df4da3f645a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc7e7fc640 1 -- 172.21.15.28:0/1127537984 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8800d5d0 con 0x55a5ccbef9f0 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc7e7fc640 1 -- 172.21.15.28:0/1127537984 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8801d070 con 0x55a5ccbef9f0 2024-09-21T05:47:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.947+0000 7fbc7e7fc640 1 -- 172.21.15.28:0/1127537984 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88021610 con 0x55a5ccbef9f0 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 msgr2=0x55a5ccfd65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 0x55a5ccfd65e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc88004400 tx=0x7fbc88005db0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 shutdown_connections 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccc10390 0x55a5ccfe0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 --2- 172.21.15.28:0/1127537984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccbef9f0 0x55a5ccfd65e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.948+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 >> 172.21.15.28:0/1127537984 conn(0x55a5ccc25460 msgr2=0x55a5ccfd5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.949+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 shutdown_connections 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.949+0000 7fbca105b740 1 -- 172.21.15.28:0/1127537984 wait complete. 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.949+0000 7fbca105b740 1 -- start start 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.949+0000 7fbca105b740 1 Processor -- start 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccbef9f0 0x55a5ccff97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbc93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44778/0 (socket says 172.21.15.28:44778) 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbc93fff640 1 -- 172.21.15.28:0/4004174468 learned_addr learned my addr 172.21.15.28:0/4004174468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccff7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ae870 con 0x55a5ccc081b0 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ae1a0 con 0x55a5ccbef9f0 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ab3e0 con 0x55a5ccc10390 2024-09-21T05:47:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbca105b740 10 client.0 _ll_register_callbacks cb 0x7fbc880042f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.950+0000 7fbc98883640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccbef9f0 0x55a5ccff97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc99084640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccff7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 -- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 msgr2=0x55a5ccff7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccff7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 -- 172.21.15.28:0/4004174468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccbef9f0 msgr2=0x55a5ccff97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccbef9f0 0x55a5ccff97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ccff7990 con 0x55a5ccc081b0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc99084640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccff7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc93fff640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbc800e4a20 tx=0x7fbc800acd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494d3b9435d14691 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc73fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc800b1c60 con 0x55a5ccc081b0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc800e4d00 con 0x55a5ccc081b0 2024-09-21T05:47:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8801d070 con 0x55a5ccc081b0 2024-09-21T05:47:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc800b6ba0 con 0x55a5ccc081b0 2024-09-21T05:47:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbca105b740 10 client.15687 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.951+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5cd0608d0 con 0x55a5ccc081b0 2024-09-21T05:47:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc800a93f0 con 0x55a5ccc081b0 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbc800d0b00 con 0x55a5ccc081b0 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 10 client.15687 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 10 client.15687 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 20 client.15687 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccc31070 0x55a5cd0630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 4 client.15687 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.952+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fbc8801d070 con 0x55a5ccc31070 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbc98883640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccc31070 0x55a5cd0630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbc73fff640 10 client.15687 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbc98883640 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccc31070 0x55a5cd0630e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=805c07eb32a9da23 server_cookie=91e164ff81d85189 in_seq=0 out_seq=0 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbc73fff640 1 -- 172.21.15.28:0/4004174468 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fbc8801d070 con 0x55a5ccc31070 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbc73fff640 10 client.15687 handle_command_reply: tid=0 2024-09-21T05:47:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.953+0000 7fbca105b740 1 client.15687 shutdown 2024-09-21T05:47:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 msgr2=0x55a5ccff6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbc800e4a20 tx=0x7fbc800acd60 comp rx=0 tx=0).stop 2024-09-21T05:47:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 shutdown_connections 2024-09-21T05:47:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a5ccc31070 0x55a5cd0630e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5ccc10390 0x55a5ccff7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5ccc081b0 0x55a5ccff6ca0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 --2- 172.21.15.28:0/4004174468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5ccbef9f0 0x55a5ccff97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 >> 172.21.15.28:0/4004174468 conn(0x55a5ccc25460 msgr2=0x55a5ccfd96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbc73fff640 0 client.15687 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 shutdown_connections 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 1 -- 172.21.15.28:0/4004174468 wait complete. 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.954+0000 7fbca105b740 20 client.15687 trim_cache size 0 max 16384 2024-09-21T05:47:17.959 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 msgr2=0x7fbc840b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 0x7fbc840b8c10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c046000 tx=0x7fbc8c048000 comp rx=0 tx=0).stop 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 msgr2=0x7fbc94198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbc80005fd0 tx=0x7fbc8000dde0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 shutdown_connections 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc94106c20 0x7fbc9419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbc840b6700 0x7fbc840b8c10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc941062d0 0x7fbc94198780 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 --2- 172.21.15.28:0/1200243750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc94101ff0 0x7fbc941981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 >> 172.21.15.28:0/1200243750 conn(0x7fbc94079e30 msgr2=0x7fbc94107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 shutdown_connections 2024-09-21T05:47:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:17.955+0000 7fbc9aae7640 1 -- 172.21.15.28:0/1200243750 wait complete. 2024-09-21T05:47:18.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:17 smithi028 ceph-mon[31227]: pgmap v333: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:18.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:17 smithi129 ceph-mon[34885]: pgmap v333: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:18.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:17 smithi181 ceph-mon[34819]: pgmap v333: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:18.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:47:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f0d078640 1 -- 172.21.15.28:0/1433291291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f040318a0 con 0x7f4f08108300 2024-09-21T05:47:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 -- 172.21.15.28:0/1433291291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f08108300 msgr2=0x7f4f08108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 --2- 172.21.15.28:0/1433291291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f08108300 0x7f4f08108710 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009900 tx=0x7f4f0402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 -- 172.21.15.28:0/1433291291 shutdown_connections 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 --2- 172.21.15.28:0/1433291291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108c50 0x7f4f0810f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 --2- 172.21.15.28:0/1433291291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f08108300 0x7f4f08108710 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 --2- 172.21.15.28:0/1433291291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f081029d0 0x7f4f08102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 -- 172.21.15.28:0/1433291291 >> 172.21.15.28:0/1433291291 conn(0x7f4f080fe410 msgr2=0x7f4f08100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.416+0000 7f4f102de640 1 -- 172.21.15.28:0/1433291291 shutdown_connections 2024-09-21T05:47:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.420+0000 7f4f102de640 1 -- 172.21.15.28:0/1433291291 wait complete. 2024-09-21T05:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.420+0000 7f4f102de640 1 Processor -- start 2024-09-21T05:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.420+0000 7f4f102de640 1 -- start start 2024-09-21T05:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 0x7f4f0819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60680/0 (socket says 172.21.15.28:60680) 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0d879640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 0x7f4f0819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0d879640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 0x7f4f0819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40158/0 (socket says 172.21.15.28:40158) 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44794/0 (socket says 172.21.15.28:44794) 2024-09-21T05:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 -- 172.21.15.28:0/336973736 learned_addr learned my addr 172.21.15.28:0/336973736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f08111170 con 0x7f4f081a2010 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f08110ff0 con 0x7f4f08108300 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081112f0 con 0x7f4f08108c50 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 -- 172.21.15.28:0/336973736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 msgr2=0x7f4f0819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 0x7f4f0819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.423+0000 7f4f0e07a640 1 -- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 msgr2=0x7f4f081a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f0e07a640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f081a2b20 con 0x7f4f08108300 2024-09-21T05:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b3e0 tx=0x7f4ef800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4eff7fe640 1 -- 172.21.15.28:0/336973736 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8015020 con 0x7f4f08108300 2024-09-21T05:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f081a2e10 con 0x7f4f08108300 2024-09-21T05:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4eff7fe640 1 -- 172.21.15.28:0/336973736 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef8011ce0 con 0x7f4f08108300 2024-09-21T05:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4eff7fe640 1 -- 172.21.15.28:0/336973736 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8010de0 con 0x7f4f08108300 2024-09-21T05:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f081a9560 con 0x7f4f08108300 2024-09-21T05:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f17253740 1 Processor -- start 2024-09-21T05:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f17253740 1 -- start start 2024-09-21T05:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f17253740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 0x55f9188919b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f17253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f918891ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f17253740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f918892430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.424+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 0x55f9188919b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 0x55f9188919b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40166/0 (socket says 172.21.15.28:40166) 2024-09-21T05:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e87b640 1 -- 172.21.15.28:0/2261846426 learned_addr learned my addr 172.21.15.28:0/2261846426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4eff7fe640 1 -- 172.21.15.28:0/336973736 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f4ef8018020 con 0x7f4f08108300 2024-09-21T05:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f9188066f0 con 0x55f9184c32f0 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f9187f5b20 con 0x55f9184cb4d0 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f9187f9930 con 0x55f9184ec1b0 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f918891ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0d879640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f918892430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 -- 172.21.15.28:0/2261846426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 msgr2=0x55f9188919b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 0x55f9188919b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 -- 172.21.15.28:0/2261846426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 msgr2=0x55f918892430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f918892430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 -- 172.21.15.28:0/2261846426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9187bb530 con 0x55f9184c32f0 2024-09-21T05:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0d879640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f918892430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f918891ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80e4560 tx=0x7f4ef80ef570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e886c6862e99326 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4eff7fe640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 0x7f4ef40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4efcbfe640 1 -- 172.21.15.28:0/2261846426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80fa070 con 0x55f9184c32f0 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 0x7f4ef40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4eff7fe640 1 -- 172.21.15.28:0/336973736 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4ef80e6df0 con 0x7f4f08108300 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4efcbfe640 1 -- 172.21.15.28:0/2261846426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef805c150 con 0x55f9184c32f0 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4efcbfe640 1 -- 172.21.15.28:0/2261846426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80fa070 con 0x55f9184c32f0 2024-09-21T05:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 msgr2=0x55f918891ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 0x7f4ef40b8b90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f000098b0 tx=0x7f4f00009600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f918891ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80e4560 tx=0x7f4ef80ef570 comp rx=0 tx=0).stop 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 shutdown_connections 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f918892430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f918891ef0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 --2- 172.21.15.28:0/2261846426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184ec1b0 0x55f9188919b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 >> 172.21.15.28:0/2261846426 conn(0x55f9184e0300 msgr2=0x55f91888f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 shutdown_connections 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- 172.21.15.28:0/2261846426 wait complete. 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 -- start start 2024-09-21T05:47:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 Processor -- start 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.430+0000 7f4f17253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184cb4d0 0x55f91889d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 0x55f9188a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44822/0 (socket says 172.21.15.28:44822) 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 -- 172.21.15.28:0/4117501736 learned_addr learned my addr 172.21.15.28:0/4117501736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0d879640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184cb4d0 0x55f91889d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f91872dab0 con 0x55f9184c32f0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f91879b730 con 0x55f9184ec1b0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f91889ffb0 con 0x55f9184cb4d0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 0x55f9188a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 10 client.0 _ll_register_callbacks cb 0x55f9187ef6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 -- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184cb4d0 msgr2=0x55f91889d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184cb4d0 0x55f91889d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 -- 172.21.15.28:0/4117501736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 msgr2=0x55f9188a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 0x55f9188a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f918557790 con 0x55f9184c32f0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 0x55f9188a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80e4610 tx=0x7f4ef80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e910ba62ee3a4ad2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80a9770 con 0x55f9184c32f0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ef805c150 con 0x55f9184c32f0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4ef80e47e0 con 0x55f9184c32f0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ef80ac890 con 0x55f9184c32f0 2024-09-21T05:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 10 client.15699 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f9185d0620 con 0x55f9184c32f0 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef805c150 con 0x55f9184c32f0 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4ef80b66b0 con 0x55f9184c32f0 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 10 client.15699 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 20 client.15699 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184aad80 0x55f9188aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 4 client.15699 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f918808640 con 0x55f9184aad80 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.435+0000 7f4f0d879640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184aad80 0x55f9188aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f0d879640 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184aad80 0x55f9188aee20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6081dceb5997c82a server_cookie=3eb135c6cdbf243 in_seq=0 out_seq=0 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4ef1ffb640 10 client.15699 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/4117501736 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f918808640 con 0x55f9184aad80 2024-09-21T05:47:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4ef1ffb640 10 client.15699 handle_command_reply: tid=0 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 client.15699 shutdown 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 msgr2=0x55f91889d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80e4610 tx=0x7f4ef80046b0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 shutdown_connections 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184aad80 0x55f9188aee20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184ec1b0 0x55f9188a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184cb4d0 0x55f91889d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 --2- 172.21.15.28:0/4117501736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184c32f0 0x55f91889d050 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 >> 172.21.15.28:0/4117501736 conn(0x55f9184e0300 msgr2=0x55f9188929e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4ef1ffb640 0 client.15699 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 shutdown_connections 2024-09-21T05:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 1 -- 172.21.15.28:0/4117501736 wait complete. 2024-09-21T05:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.440+0000 7f4f17253740 20 client.15699 trim_cache size 0 max 16384 2024-09-21T05:47:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 Processor -- start 2024-09-21T05:47:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 -- start start 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184e0300 0x55f91889a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f91889a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081b1fb0 con 0x55f9184cb4d0 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081b3af0 con 0x55f9184e0300 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.470+0000 7f4f17253740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081b2680 con 0x55f9184c32f0 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44836/0 (socket says 172.21.15.28:44836) 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 -- 172.21.15.28:0/1817774817 learned_addr learned my addr 172.21.15.28:0/1817774817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0d879640 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f91889a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184e0300 0x55f91889a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 -- 172.21.15.28:0/1817774817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 msgr2=0x55f91889a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f91889a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 -- 172.21.15.28:0/1817774817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184e0300 msgr2=0x55f91889a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184e0300 0x55f91889a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 -- 172.21.15.28:0/1817774817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f91889e310 con 0x55f9184cb4d0 2024-09-21T05:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.471+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4f00013d80 tx=0x7f4f0000b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef93306f9258d69c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.472+0000 7f4ef3fff640 1 -- 172.21.15.28:0/1817774817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0002c070 con 0x55f9184cb4d0 2024-09-21T05:47:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.472+0000 7f4ef3fff640 1 -- 172.21.15.28:0/1817774817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f0001c910 con 0x55f9184cb4d0 2024-09-21T05:47:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.472+0000 7f4ef3fff640 1 -- 172.21.15.28:0/1817774817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f000278e0 con 0x55f9184cb4d0 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 -- 172.21.15.28:0/1817774817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 msgr2=0x55f9188b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4f00013d80 tx=0x7f4f0000b000 comp rx=0 tx=0).stop 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 -- 172.21.15.28:0/1817774817 shutdown_connections 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184cb4d0 0x55f9188b43c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f91889a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 --2- 172.21.15.28:0/1817774817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184e0300 0x55f91889a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 -- 172.21.15.28:0/1817774817 >> 172.21.15.28:0/1817774817 conn(0x55f9188a9750 msgr2=0x55f91889b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 -- 172.21.15.28:0/1817774817 shutdown_connections 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.473+0000 7f4f17253740 1 -- 172.21.15.28:0/1817774817 wait complete. 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.474+0000 7f4f17253740 1 -- start start 2024-09-21T05:47:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.474+0000 7f4f17253740 1 Processor -- start 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f9188a1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f9188a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081b3420 con 0x55f9184e0300 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081b2d50 con 0x55f9184cb4d0 2024-09-21T05:47:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081aff90 con 0x55f9184c32f0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f9188a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e07a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f9188a1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40204/0 (socket says 172.21.15.28:40204) 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44842/0 (socket says 172.21.15.28:44842) 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e07a640 1 -- 172.21.15.28:0/2025555402 learned_addr learned my addr 172.21.15.28:0/2025555402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0d879640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f9188a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f17253740 10 client.0 _ll_register_callbacks cb 0x7f4ef80f8070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 -- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 msgr2=0x55f9188a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f9188a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 -- 172.21.15.28:0/2025555402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 msgr2=0x55f9188a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f9188a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.475+0000 7f4f0e87b640 1 -- 172.21.15.28:0/2025555402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f918921310 con 0x55f9184e0300 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4f0e87b640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009f20 tx=0x7f4f00009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0ac5ad2a45596e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4ef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0002c040 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4f00013db0 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f0001c910 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4f0001cac0 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4f17253740 10 client.15705 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.476+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f9189223d0 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f0001c910 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4f0002f070 con 0x55f9184e0300 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 10 client.15705 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 20 client.15705 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184ec1b0 0x55f918924a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 4 client.15705 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9188040f0 con 0x55f9184ec1b0 2024-09-21T05:47:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.477+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184ec1b0 0x55f918924a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.478+0000 7f4ef1ffb640 10 client.15705 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.478+0000 7f4f0e07a640 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184ec1b0 0x55f918924a00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dd9678b40d565a1 server_cookie=cd5afd476bec304d in_seq=0 out_seq=0 2024-09-21T05:47:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4ef1ffb640 1 -- 172.21.15.28:0/2025555402 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9188040f0 con 0x55f9184ec1b0 2024-09-21T05:47:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4ef1ffb640 10 client.15705 handle_command_reply: tid=0 2024-09-21T05:47:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 client.15705 shutdown 2024-09-21T05:47:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 msgr2=0x55f918920dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009f20 tx=0x7f4f00009430 comp rx=0 tx=0).stop 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 shutdown_connections 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f9184ec1b0 0x55f918924a00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f9184e0300 0x55f918920dd0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f9184cb4d0 0x55f9188a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 --2- 172.21.15.28:0/2025555402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9184c32f0 0x55f9188a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4ef1ffb640 0 client.15705 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 >> 172.21.15.28:0/2025555402 conn(0x55f9188a9750 msgr2=0x55f91888dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 shutdown_connections 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.486+0000 7f4f17253740 1 -- 172.21.15.28:0/2025555402 wait complete. 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f17253740 20 client.15705 trim_cache size 0 max 16384 2024-09-21T05:47:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:47:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:47:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:47:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:47:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 msgr2=0x7f4ef40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 0x7f4ef40b8b90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f000098b0 tx=0x7f4f00009600 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 msgr2=0x7f4f0819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b3e0 tx=0x7f4ef800b8a0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 shutdown_connections 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4ef40b6680 0x7f4ef40b8b90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f081a2010 0x7f4f081a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f08108c50 0x7f4f0819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 --2- 172.21.15.28:0/336973736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f08108300 0x7f4f0819c7e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 >> 172.21.15.28:0/336973736 conn(0x7f4f080fe410 msgr2=0x7f4f08106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 shutdown_connections 2024-09-21T05:47:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:19.487+0000 7f4f102de640 1 -- 172.21.15.28:0/336973736 wait complete. 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 -- 172.21.15.28:0/2429363872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c102a60 msgr2=0x7f518c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 --2- 172.21.15.28:0/2429363872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c102a60 0x7f518c102e70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5188009900 tx=0x7f518802a410 comp rx=0 tx=0).stop 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 -- 172.21.15.28:0/2429363872 shutdown_connections 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 --2- 172.21.15.28:0/2429363872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c108cf0 0x7f518c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 --2- 172.21.15.28:0/2429363872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 --2- 172.21.15.28:0/2429363872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c102a60 0x7f518c102e70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 -- 172.21.15.28:0/2429363872 >> 172.21.15.28:0/2429363872 conn(0x7f518c079730 msgr2=0x7f518c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 -- 172.21.15.28:0/2429363872 shutdown_connections 2024-09-21T05:47:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.649+0000 7f51935f5640 1 -- 172.21.15.28:0/2429363872 wait complete. 2024-09-21T05:47:19.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.650+0000 7f51935f5640 1 Processor -- start 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.650+0000 7f51935f5640 1 -- start start 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 0x7f518c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c199b20 0x7f518c19e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f5191391640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f5190b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 0x7f518c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f5191391640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60722/0 (socket says 172.21.15.28:60722) 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f5191391640 1 -- 172.21.15.28:0/1927266998 learned_addr learned my addr 172.21.15.28:0/1927266998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f518c112960 con 0x7f518c108cf0 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f518c1127e0 con 0x7f518c1083a0 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.651+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f518c112ae0 con 0x7f518c199b20 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191b92640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c199b20 0x7f518c19e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 -- 172.21.15.28:0/1927266998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c199b20 msgr2=0x7f518c19e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c199b20 0x7f518c19e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 -- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 msgr2=0x7f518c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 0x7f518c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c19e790 con 0x7f518c1083a0 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5190b90640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 0x7f518c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:19.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f5191391640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5188006010 tx=0x7f51880042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f517e7fc640 1 -- 172.21.15.28:0/1927266998 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518803f030 con 0x7f518c1083a0 2024-09-21T05:47:19.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.652+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c19ea20 con 0x7f518c1083a0 2024-09-21T05:47:19.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.653+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f518c19ef00 con 0x7f518c1083a0 2024-09-21T05:47:19.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.653+0000 7f517e7fc640 1 -- 172.21.15.28:0/1927266998 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5188003cb0 con 0x7f518c1083a0 2024-09-21T05:47:19.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.653+0000 7f517e7fc640 1 -- 172.21.15.28:0/1927266998 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188035070 con 0x7f518c1083a0 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 Processor -- start 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 -- start start 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 0x55c2836c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 0x55c2836c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c283636ea0 con 0x55c2832f8390 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c2836262d0 con 0x55c2832f01b0 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.656+0000 7f5199b69740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c28362a0e0 con 0x55c283319070 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5190b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5191b92640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 0x55c2836c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f517e7fc640 1 -- 172.21.15.28:0/1927266998 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5188042030 con 0x7f518c1083a0 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5191b92640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 0x55c2836c06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40230/0 (socket says 172.21.15.28:40230) 2024-09-21T05:47:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5191b92640 1 -- 172.21.15.28:0/1958009859 learned_addr learned my addr 172.21.15.28:0/1958009859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5191391640 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 0x55c2836c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5190b90640 1 -- 172.21.15.28:0/1958009859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 msgr2=0x55c2836c06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5190b90640 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 0x55c2836c06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.657+0000 7f5190b90640 1 -- 172.21.15.28:0/1958009859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 msgr2=0x55c2836c0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.658+0000 7f5190b90640 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 0x55c2836c0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.658+0000 7f5190b90640 1 -- 172.21.15.28:0/1958009859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c283384650 con 0x55c2832f8390 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.658+0000 7f5191391640 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 0x55c2836c0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.658+0000 7f5190b90640 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836c1120 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f518c06aaa0 tx=0x7f5180010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517a45399ec5c94b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f517e7fc640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 0x7f51780b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f517e7fc640 1 -- 172.21.15.28:0/1927266998 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f51881077b0 con 0x7f518c1083a0 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f51777fe640 1 -- 172.21.15.28:0/1958009859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51800117c0 con 0x55c2832f8390 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f51777fe640 1 -- 172.21.15.28:0/1958009859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5180011e10 con 0x55c2832f8390 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f51777fe640 1 -- 172.21.15.28:0/1958009859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518001a380 con 0x55c2832f8390 2024-09-21T05:47:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.659+0000 7f5191b92640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 0x7f51780b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 -- 172.21.15.28:0/1958009859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 msgr2=0x55c2836c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5191b92640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 0x7f51780b8c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5184010590 tx=0x7f5184007520 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836c1120 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f518c06aaa0 tx=0x7f5180010860 comp rx=0 tx=0).stop 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 -- 172.21.15.28:0/1958009859 shutdown_connections 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836c1120 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f01b0 0x55c2836c0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 --2- 172.21.15.28:0/1958009859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c283319070 0x55c2836c06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 -- 172.21.15.28:0/1958009859 >> 172.21.15.28:0/1958009859 conn(0x55c28330d460 msgr2=0x55c2836be590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 -- 172.21.15.28:0/1958009859 shutdown_connections 2024-09-21T05:47:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.660+0000 7f5199b69740 1 -- 172.21.15.28:0/1958009859 wait complete. 2024-09-21T05:47:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 -- start start 2024-09-21T05:47:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 Processor -- start 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c283319070 0x55c2836d8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c283487700 con 0x55c2832f8390 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c2835c1160 con 0x55c283319070 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.661+0000 7f5199b69740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c2836cee80 con 0x55c2832f01b0 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5199b69740 10 client.0 _ll_register_callbacks cb 0x55c2835da9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5190b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5190b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44886/0 (socket says 172.21.15.28:44886) 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5190b90640 1 -- 172.21.15.28:0/1701391319 learned_addr learned my addr 172.21.15.28:0/1701391319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191b92640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c283319070 0x55c2836d8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 -- 172.21.15.28:0/1701391319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c283319070 msgr2=0x55c2836d8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c283319070 0x55c2836d8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 -- 172.21.15.28:0/1701391319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 msgr2=0x55c2836cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 -- 172.21.15.28:0/1701391319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2835eec90 con 0x55c2832f01b0 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5191391640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836cbda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5188104ec0 tx=0x7f51880cd500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80a4139fa57a9a1a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.662+0000 7f5174ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188049070 con 0x55c2832f01b0 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5188107be0 con 0x55c2832f01b0 2024-09-21T05:47:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5180011e10 con 0x55c2832f01b0 2024-09-21T05:47:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5188003e60 con 0x55c2832f01b0 2024-09-21T05:47:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 10 client.25393 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c2836dab70 con 0x55c2832f01b0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51880cd180 con 0x55c2832f01b0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f518803bbb0 con 0x55c2832f01b0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 10 client.25393 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 10 client.25393 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 20 client.25393 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c2832d79f0 0x55c2836de080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 4 client.25393 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.663+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5180011e10 con 0x55c2832d79f0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.664+0000 7f5190b90640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c2832d79f0 0x55c2836de080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.664+0000 7f5174ff9640 10 client.25393 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.664+0000 7f5190b90640 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c2832d79f0 0x55c2836de080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39b6dd9e4244e9d1 server_cookie=63b3909534e5b895 in_seq=0 out_seq=0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5174ff9640 1 -- 172.21.15.28:0/1701391319 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f5180011e10 con 0x55c2832d79f0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5174ff9640 10 client.25393 handle_command_reply: tid=0 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 client.25393 shutdown 2024-09-21T05:47:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 msgr2=0x55c2836cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836cbda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5188104ec0 tx=0x7f51880cd500 comp rx=0 tx=0).stop 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 shutdown_connections 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c2832d79f0 0x55c2836de080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c283319070 0x55c2836d8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.665+0000 7f5174ff9640 0 client.25393 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 1 --2- 172.21.15.28:0/1701391319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836cbda0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 >> 172.21.15.28:0/1701391319 conn(0x55c28330d460 msgr2=0x55c2836c3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 shutdown_connections 2024-09-21T05:47:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 1 -- 172.21.15.28:0/1701391319 wait complete. 2024-09-21T05:47:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.666+0000 7f5199b69740 20 client.25393 trim_cache size 0 max 16384 2024-09-21T05:47:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.679+0000 7f5199b69740 1 Processor -- start 2024-09-21T05:47:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.679+0000 7f5199b69740 1 -- start start 2024-09-21T05:47:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 0x55c2836c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f8390 0x55c2836bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40256/0 (socket says 172.21.15.28:40256) 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 -- 172.21.15.28:0/2257662610 learned_addr learned my addr 172.21.15.28:0/2257662610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f517000e740 con 0x55c2832f01b0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5170010280 con 0x55c2832f8390 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f517000ee10 con 0x55c2832d79f0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5190b90640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 0x55c2836c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191b92640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f8390 0x55c2836bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 -- 172.21.15.28:0/2257662610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f8390 msgr2=0x55c2836bdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f8390 0x55c2836bdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.680+0000 7f5191391640 1 -- 172.21.15.28:0/2257662610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 msgr2=0x55c2836c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5191391640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 0x55c2836c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5191391640 1 -- 172.21.15.28:0/2257662610 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2836c14b0 con 0x55c2832d79f0 2024-09-21T05:47:19.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5190b90640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 0x55c2836c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5191391640 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5188005940 tx=0x7f5188005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee4ed2e138ed011 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5176ffd640 1 -- 172.21.15.28:0/2257662610 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188049070 con 0x55c2832d79f0 2024-09-21T05:47:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5176ffd640 1 -- 172.21.15.28:0/2257662610 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51880d6fe0 con 0x55c2832d79f0 2024-09-21T05:47:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.681+0000 7f5176ffd640 1 -- 172.21.15.28:0/2257662610 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518803b660 con 0x55c2832d79f0 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 msgr2=0x55c2836cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5188005940 tx=0x7f5188005f30 comp rx=0 tx=0).stop 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 shutdown_connections 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832f8390 0x55c2836bdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f01b0 0x55c2836c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 --2- 172.21.15.28:0/2257662610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832d79f0 0x55c2836cdc80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 >> 172.21.15.28:0/2257662610 conn(0x55c28330d460 msgr2=0x55c2836c16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 shutdown_connections 2024-09-21T05:47:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- 172.21.15.28:0/2257662610 wait complete. 2024-09-21T05:47:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 -- start start 2024-09-21T05:47:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.683+0000 7f5199b69740 1 Processor -- start 2024-09-21T05:47:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f517000fbb0 con 0x55c2832f8390 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f517000f4e0 con 0x55c2832d79f0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f517000c720 con 0x55c2832f01b0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5199b69740 10 client.0 _ll_register_callbacks cb 0x7f51880c9510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5191391640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5191391640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60768/0 (socket says 172.21.15.28:60768) 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.684+0000 7f5191391640 1 -- 172.21.15.28:0/650897393 learned_addr learned my addr 172.21.15.28:0/650897393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5190b90640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191b92640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 -- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 msgr2=0x55c2836d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 -- 172.21.15.28:0/650897393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 msgr2=0x55c2836d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 -- 172.21.15.28:0/650897393 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c283747a80 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191b92640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5191391640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51880c9d60 tx=0x7f5188005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6da2ecf81087f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5174ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188049070 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5188003a00 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.685+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51880d6fe0 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.686+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f51880d20a0 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.686+0000 7f5199b69740 10 client.25340 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.686+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c283747a80 con 0x55c2832d79f0 2024-09-21T05:47:19.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.686+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51880d62c0 con 0x55c2832d79f0 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.686+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f51880cc340 con 0x55c2832d79f0 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 10 client.25340 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 10 client.25340 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 20 client.25340 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c283319070 0x55c28374afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 4 client.25340 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f51880d6fe0 con 0x55c283319070 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.687+0000 7f5190b90640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c283319070 0x55c28374afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5174ff9640 10 client.25340 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5190b90640 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c283319070 0x55c28374afe0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8274c9777910817 server_cookie=dffd983184330ab4 in_seq=0 out_seq=0 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5174ff9640 1 -- 172.21.15.28:0/650897393 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f51880d6fe0 con 0x55c283319070 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5174ff9640 10 client.25340 handle_command_reply: tid=0 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 client.25340 shutdown 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 msgr2=0x55c2836d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51880c9d60 tx=0x7f5188005ec0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 shutdown_connections 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55c283319070 0x55c28374afe0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2832f8390 0x55c2836d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2832f01b0 0x55c2836d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5199b69740 1 --2- 172.21.15.28:0/650897393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2832d79f0 0x55c2836d0110 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.688+0000 7f5174ff9640 0 client.25340 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.689+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 >> 172.21.15.28:0/650897393 conn(0x55c28330d460 msgr2=0x55c2836c2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.689+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 shutdown_connections 2024-09-21T05:47:19.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.689+0000 7f5199b69740 1 -- 172.21.15.28:0/650897393 wait complete. 2024-09-21T05:47:19.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.689+0000 7f5199b69740 20 client.25340 trim_cache size 0 max 16384 2024-09-21T05:47:19.694 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 msgr2=0x7f51780b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 0x7f51780b8c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5184010590 tx=0x7f5184007520 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 msgr2=0x7f518c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5188006010 tx=0x7f51880042f0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 shutdown_connections 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f518c199b20 0x7f518c19e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51780b6780 0x7f51780b8c90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c108cf0 0x7f518c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 --2- 172.21.15.28:0/1927266998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f518c1083a0 0x7f518c1990a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 >> 172.21.15.28:0/1927266998 conn(0x7f518c079730 msgr2=0x7f518c101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 shutdown_connections 2024-09-21T05:47:19.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:19.690+0000 7f51935f5640 1 -- 172.21.15.28:0/1927266998 wait complete. 2024-09-21T05:47:20.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:19 smithi129 ceph-mon[34885]: pgmap v334: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-09-21T05:47:20.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:19 smithi028 ceph-mon[31227]: pgmap v334: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-09-21T05:47:20.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:19 smithi181 ceph-mon[34819]: pgmap v334: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-09-21T05:47:20.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:47:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.097+0000 7f92077b2640 1 -- 172.21.15.28:0/1158084774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9200108d10 msgr2=0x7f920010f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.097+0000 7f92077b2640 1 --2- 172.21.15.28:0/1158084774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9200108d10 0x7f920010f540 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009ca0 tx=0x7f91f802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.097+0000 7f92077b2640 1 -- 172.21.15.28:0/1158084774 shutdown_connections 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 --2- 172.21.15.28:0/1158084774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9200108d10 0x7f920010f540 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 --2- 172.21.15.28:0/1158084774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92001083c0 0x7f92001087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 --2- 172.21.15.28:0/1158084774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 0x7f9200102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 -- 172.21.15.28:0/1158084774 >> 172.21.15.28:0/1158084774 conn(0x7f9200079740 msgr2=0x7f9200079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 -- 172.21.15.28:0/1158084774 shutdown_connections 2024-09-21T05:47:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.098+0000 7f92077b2640 1 -- 172.21.15.28:0/1158084774 wait complete. 2024-09-21T05:47:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.099+0000 7f92077b2640 1 Processor -- start 2024-09-21T05:47:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.099+0000 7f92077b2640 1 -- start start 2024-09-21T05:47:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 0x7f9200198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 0x7f9200198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92001118c0 con 0x7f92001083c0 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9200111740 con 0x7f9200102a80 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.100+0000 7f92077b2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9200111a40 con 0x7f9200108d10 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f920554e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 0x7f9200198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40274/0 (socket says 172.21.15.28:40274) 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 -- 172.21.15.28:0/2740665010 learned_addr learned my addr 172.21.15.28:0/2740665010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9204d4d640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 0x7f9200198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 -- 172.21.15.28:0/2740665010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 msgr2=0x7f9200198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 0x7f9200198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 -- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 msgr2=0x7f9200198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 0x7f9200198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9205d4f640 1 -- 172.21.15.28:0/2740665010 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920019e200 con 0x7f9200108d10 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.101+0000 7f9204d4d640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 0x7f9200198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.102+0000 7f9205d4f640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009430 tx=0x7f91f80317c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.102+0000 7f91f67fc640 1 -- 172.21.15.28:0/2740665010 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f8003b70 con 0x7f9200108d10 2024-09-21T05:47:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.102+0000 7f91f67fc640 1 -- 172.21.15.28:0/2740665010 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91f8003d10 con 0x7f9200108d10 2024-09-21T05:47:21.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.102+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f920019e490 con 0x7f9200108d10 2024-09-21T05:47:21.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.102+0000 7f91f67fc640 1 -- 172.21.15.28:0/2740665010 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f802fa30 con 0x7f9200108d10 2024-09-21T05:47:21.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.103+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92001a4c50 con 0x7f9200108d10 2024-09-21T05:47:21.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.107+0000 7f91f67fc640 1 -- 172.21.15.28:0/2740665010 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f91f802fbd0 con 0x7f9200108d10 2024-09-21T05:47:21.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.107+0000 7f920dd26740 1 Processor -- start 2024-09-21T05:47:21.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 -- start start 2024-09-21T05:47:21.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d71523ea0 con 0x561d711dd1b0 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d715132d0 con 0x561d711e5390 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f920dd26740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d715170e0 con 0x561d71206070 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.108+0000 7f91f67fc640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 0x7f91ec0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f91f67fc640 1 -- 172.21.15.28:0/2740665010 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f91f8108930 con 0x7f9200108d10 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9204d4d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9204d4d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60790/0 (socket says 172.21.15.28:60790) 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9204d4d640 1 -- 172.21.15.28:0/3862942819 learned_addr learned my addr 172.21.15.28:0/3862942819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:21.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9204d4d640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9205d4f640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 0x7f91ec0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f920554e640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.109+0000 7f9205d4f640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 0x7f91ec0b8c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f91f8005e30 tx=0x7f91f80095c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 -- 172.21.15.28:0/3862942819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 msgr2=0x561d715add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 -- 172.21.15.28:0/3862942819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 msgr2=0x561d715ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 -- 172.21.15.28:0/3862942819 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d71271650 con 0x561d71206070 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f920554e640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.110+0000 7f9204d4d640 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715ad2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91f0003ba0 tx=0x7f91f0011040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56dfdfabac2d0b6e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.111+0000 7f91eb7fe640 1 -- 172.21.15.28:0/3862942819 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f001ec50 con 0x561d71206070 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.111+0000 7f91eb7fe640 1 -- 172.21.15.28:0/3862942819 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91f000f4d0 con 0x561d71206070 2024-09-21T05:47:21.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.111+0000 7f91eb7fe640 1 -- 172.21.15.28:0/3862942819 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f000ebf0 con 0x561d71206070 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 -- 172.21.15.28:0/3862942819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 msgr2=0x561d715ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715ad2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91f0003ba0 tx=0x7f91f0011040 comp rx=0 tx=0).stop 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 -- 172.21.15.28:0/3862942819 shutdown_connections 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 --2- 172.21.15.28:0/3862942819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715ad2c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 -- 172.21.15.28:0/3862942819 >> 172.21.15.28:0/3862942819 conn(0x561d711fa460 msgr2=0x561d715ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 -- 172.21.15.28:0/3862942819 shutdown_connections 2024-09-21T05:47:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.112+0000 7f920dd26740 1 -- 172.21.15.28:0/3862942819 wait complete. 2024-09-21T05:47:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.113+0000 7f920dd26740 1 -- start start 2024-09-21T05:47:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.113+0000 7f920dd26740 1 Processor -- start 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d71374700 con 0x561d711dd1b0 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d714ae160 con 0x561d711e5390 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d715bba40 con 0x561d71206070 2024-09-21T05:47:21.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920dd26740 10 client.0 _ll_register_callbacks cb 0x561d714c79e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60798/0 (socket says 172.21.15.28:60798) 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 -- 172.21.15.28:0/224774877 learned_addr learned my addr 172.21.15.28:0/224774877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f9204d4d640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 -- 172.21.15.28:0/224774877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 msgr2=0x561d715c4d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c4d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.114+0000 7f920554e640 1 -- 172.21.15.28:0/224774877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 msgr2=0x561d715b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f920554e640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f920554e640 1 -- 172.21.15.28:0/224774877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d714dbc90 con 0x561d711e5390 2024-09-21T05:47:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f9204d4d640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f920554e640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00a840 tx=0x7f91fc00ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce200570f80dc996 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f91e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc013ba0 con 0x561d711e5390 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.115+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f91fc013d40 con 0x561d711e5390 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.116+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91f000f4d0 con 0x561d711e5390 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.116+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f91fc00ae30 con 0x561d711e5390 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.116+0000 7f920dd26740 10 client.25352 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.116+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d715c78b0 con 0x561d711e5390 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.116+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91fc0178e0 con 0x561d711e5390 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f91fc01f6a0 con 0x561d711e5390 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 10 client.25352 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 10 client.25352 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 20 client.25352 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d711c49f0 0x561d715cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 4 client.25352 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.117+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f91f000f4d0 con 0x561d711c49f0 2024-09-21T05:47:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.118+0000 7f9204d4d640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d711c49f0 0x561d715cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.118+0000 7f91e8ff9640 10 client.25352 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.118+0000 7f9204d4d640 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d711c49f0 0x561d715cae40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c7a8bd4f0fcd69f server_cookie=4cb56c7902f49b42 in_seq=0 out_seq=0 2024-09-21T05:47:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.119+0000 7f91e8ff9640 1 -- 172.21.15.28:0/224774877 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f91f000f4d0 con 0x561d711c49f0 2024-09-21T05:47:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.119+0000 7f91e8ff9640 10 client.25352 handle_command_reply: tid=0 2024-09-21T05:47:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.119+0000 7f920dd26740 1 client.25352 shutdown 2024-09-21T05:47:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.119+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 msgr2=0x561d715b8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.119+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00a840 tx=0x7f91fc00ad00 comp rx=0 tx=0).stop 2024-09-21T05:47:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 shutdown_connections 2024-09-21T05:47:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d711c49f0 0x561d715cae40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715b8fa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 --2- 172.21.15.28:0/224774877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 >> 172.21.15.28:0/224774877 conn(0x561d711fa460 msgr2=0x561d715b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f91e8ff9640 0 client.25352 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 shutdown_connections 2024-09-21T05:47:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 1 -- 172.21.15.28:0/224774877 wait complete. 2024-09-21T05:47:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.120+0000 7f920dd26740 20 client.25352 trim_cache size 0 max 16384 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 Processor -- start 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 -- start start 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92001ad420 con 0x561d711dd1b0 2024-09-21T05:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92001aef60 con 0x561d711e5390 2024-09-21T05:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.139+0000 7f920dd26740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92001adaf0 con 0x561d71206070 2024-09-21T05:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f9204d4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f9204d4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44956/0 (socket says 172.21.15.28:44956) 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f9204d4d640 1 -- 172.21.15.28:0/3915511302 learned_addr learned my addr 172.21.15.28:0/3915511302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f9204d4d640 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 -- 172.21.15.28:0/3915511302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 msgr2=0x561d715c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 -- 172.21.15.28:0/3915511302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 msgr2=0x561d715b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 -- 172.21.15.28:0/3915511302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d715b3670 con 0x561d711e5390 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.140+0000 7f920554e640 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715c4c60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00a840 tx=0x7f91fc00a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8390cb2e9fd5d970 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.141+0000 7f91eaffd640 1 -- 172.21.15.28:0/3915511302 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc035070 con 0x561d711e5390 2024-09-21T05:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.141+0000 7f91eaffd640 1 -- 172.21.15.28:0/3915511302 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91fc01f8b0 con 0x561d711e5390 2024-09-21T05:47:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.141+0000 7f91eaffd640 1 -- 172.21.15.28:0/3915511302 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc02c040 con 0x561d711e5390 2024-09-21T05:47:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- 172.21.15.28:0/3915511302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 msgr2=0x561d715c4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715c4c60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00a840 tx=0x7f91fc00a7d0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- 172.21.15.28:0/3915511302 shutdown_connections 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d71206070 0x561d715c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d711e5390 0x561d715c4c60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 --2- 172.21.15.28:0/3915511302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711dd1b0 0x561d715b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- 172.21.15.28:0/3915511302 >> 172.21.15.28:0/3915511302 conn(0x561d711fa460 msgr2=0x561d715b8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- 172.21.15.28:0/3915511302 shutdown_connections 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- 172.21.15.28:0/3915511302 wait complete. 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.142+0000 7f920dd26740 1 -- start start 2024-09-21T05:47:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 Processor -- start 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d711dd1b0 0x561d715c1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d71206070 0x561d715bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92001ae890 con 0x561d711e5390 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92001ae1c0 con 0x561d71206070 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92001ab400 con 0x561d711dd1b0 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920554e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f9204d4d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d711dd1b0 0x561d715c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920554e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44960/0 (socket says 172.21.15.28:44960) 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920dd26740 10 client.0 _ll_register_callbacks cb 0x7f91fc00b880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.143+0000 7f920554e640 1 -- 172.21.15.28:0/1849330185 learned_addr learned my addr 172.21.15.28:0/1849330185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f9204d4d640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d71206070 0x561d715bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 -- 172.21.15.28:0/1849330185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d711dd1b0 msgr2=0x561d715c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d711dd1b0 0x561d715c1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 -- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d71206070 msgr2=0x561d715bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d71206070 0x561d715bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 -- 172.21.15.28:0/1849330185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d715bd390 con 0x561d711e5390 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f920554e640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f91fc000c00 tx=0x7f91fc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abb18e1b7d196df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.144+0000 7f91e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc035070 con 0x561d711e5390 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f91fc0139a0 con 0x561d711e5390 2024-09-21T05:47:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91fc01f8b0 con 0x561d711e5390 2024-09-21T05:47:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f91fc02c070 con 0x561d711e5390 2024-09-21T05:47:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f920dd26740 10 client.15738 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d71635630 con 0x561d711e5390 2024-09-21T05:47:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91fc01f8b0 con 0x561d711e5390 2024-09-21T05:47:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.145+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f91fc0165c0 con 0x561d711e5390 2024-09-21T05:47:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 10 client.15738 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 10 client.15738 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 20 client.15738 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d71199da0 0x561d71637c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 4 client.15738 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d71525df0 con 0x561d71199da0 2024-09-21T05:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f9204d4d640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d71199da0 0x561d71637c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f91e8ff9640 10 client.15738 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.146+0000 7f9204d4d640 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d71199da0 0x561d71637c80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=578c798930cddea6 server_cookie=3d163083033dd0d3 in_seq=0 out_seq=0 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f91e8ff9640 1 -- 172.21.15.28:0/1849330185 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d71525df0 con 0x561d71199da0 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f91e8ff9640 10 client.15738 handle_command_reply: tid=0 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 client.15738 shutdown 2024-09-21T05:47:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 msgr2=0x561d715bc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f91fc000c00 tx=0x7f91fc00be10 comp rx=0 tx=0).stop 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 shutdown_connections 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x561d71199da0 0x561d71637c80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d71206070 0x561d715bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d711e5390 0x561d715bc810 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f91e8ff9640 0 client.15738 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 --2- 172.21.15.28:0/1849330185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d711dd1b0 0x561d715c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 >> 172.21.15.28:0/1849330185 conn(0x561d711fa460 msgr2=0x561d715b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 shutdown_connections 2024-09-21T05:47:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.147+0000 7f920dd26740 1 -- 172.21.15.28:0/1849330185 wait complete. 2024-09-21T05:47:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f920dd26740 20 client.15738 trim_cache size 0 max 16384 2024-09-21T05:47:21.157 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 msgr2=0x7f91ec0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 0x7f91ec0b8c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f91f8005e30 tx=0x7f91f80095c0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 msgr2=0x7f920019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009430 tx=0x7f91f80317c0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 shutdown_connections 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9200108d10 0x7f920019dad0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91ec0b6700 0x7f91ec0b8c10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92001083c0 0x7f9200198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.148+0000 7f92077b2640 1 --2- 172.21.15.28:0/2740665010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9200102a80 0x7f9200198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.149+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 >> 172.21.15.28:0/2740665010 conn(0x7f9200079740 msgr2=0x7f9200104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.149+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 shutdown_connections 2024-09-21T05:47:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:21.149+0000 7f92077b2640 1 -- 172.21.15.28:0/2740665010 wait complete. 2024-09-21T05:47:21.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:47:22.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:21 smithi129 ceph-mon[34885]: pgmap v335: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-21T05:47:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:22.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:21 smithi028 ceph-mon[31227]: pgmap v335: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-21T05:47:22.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:21 smithi181 ceph-mon[34819]: pgmap v335: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-21T05:47:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.675+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/128096135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f4108cf0 msgr2=0x7fb6f410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.675+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/128096135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f4108cf0 0x7fb6f410f520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009ca0 tx=0x7fb6ec02a8e0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/128096135 shutdown_connections 2024-09-21T05:47:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/128096135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f4108cf0 0x7fb6f410f520 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/128096135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f41083a0 0x7fb6f41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/128096135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4102a60 0x7fb6f4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/128096135 >> 172.21.15.28:0/128096135 conn(0x7fb6f4079730 msgr2=0x7fb6f4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/128096135 shutdown_connections 2024-09-21T05:47:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.676+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/128096135 wait complete. 2024-09-21T05:47:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.677+0000 7fb6fc1a7640 1 Processor -- start 2024-09-21T05:47:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.677+0000 7fb6fc1a7640 1 -- start start 2024-09-21T05:47:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 0x7fb6f410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4108cf0 0x7fb6f41100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113ac0 con 0x7fb6f41083a0 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113940 con 0x7fb6f4108cf0 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fc1a7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f4113c40 con 0x7fb6f4102a60 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40314/0 (socket says 172.21.15.28:40314) 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3870968676 learned_addr learned my addr 172.21.15.28:0/3870968676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9742640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 0x7fb6f410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6fa744640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4108cf0 0x7fb6f41100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3870968676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4108cf0 msgr2=0x7fb6f41100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4108cf0 0x7fb6f41100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 msgr2=0x7fb6f410fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 0x7fb6f410fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.678+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3870968676 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f4110910 con 0x7fb6f4102a60 2024-09-21T05:47:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6f9742640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 0x7fb6f410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000d950 tx=0x7fb6f000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6eaffd640 1 -- 172.21.15.28:0/3870968676 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0017030 con 0x7fb6f4102a60 2024-09-21T05:47:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6eaffd640 1 -- 172.21.15.28:0/3870968676 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f0004020 con 0x7fb6f4102a60 2024-09-21T05:47:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f41acbe0 con 0x7fb6f4102a60 2024-09-21T05:47:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6eaffd640 1 -- 172.21.15.28:0/3870968676 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0003c50 con 0x7fb6f4102a60 2024-09-21T05:47:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.679+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f41ad0d0 con 0x7fb6f4102a60 2024-09-21T05:47:22.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.683+0000 7fb70271b740 1 Processor -- start 2024-09-21T05:47:22.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.683+0000 7fb70271b740 1 -- start start 2024-09-21T05:47:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.684+0000 7fb6eaffd640 1 -- 172.21.15.28:0/3870968676 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb6f001a070 con 0x7fb6f4102a60 2024-09-21T05:47:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.684+0000 7fb70271b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe85290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.684+0000 7fb70271b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.684+0000 7fb70271b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe85d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.684+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45006/0 (socket says 172.21.15.28:45006) 2024-09-21T05:47:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6fa744640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe85290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6fa744640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe85290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:40324/0 (socket says 172.21.15.28:40324) 2024-09-21T05:47:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6f9f43640 1 -- 172.21.15.28:0/4272737300 learned_addr learned my addr 172.21.15.28:0/4272737300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645fbdfbea0 con 0x5645fbab51b0 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5645fbdeb2d0 con 0x5645fbabd390 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6f9742640 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe85d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5645fbdef0e0 con 0x5645fbade070 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.685+0000 7fb6eaffd640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 0x7fb6e00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6fa744640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 0x7fb6e00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 -- 172.21.15.28:0/4272737300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 msgr2=0x5645fbe85290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe85290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 -- 172.21.15.28:0/4272737300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 msgr2=0x5645fbe85d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe85d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 -- 172.21.15.28:0/4272737300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645fbb49650 con 0x5645fbab51b0 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.686+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00acf80 tx=0x7fb6f00ddd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40867c757891219a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.687+0000 7fb6dffff640 1 -- 172.21.15.28:0/4272737300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00ac9f0 con 0x5645fbab51b0 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.687+0000 7fb6dffff640 1 -- 172.21.15.28:0/4272737300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f00acd90 con 0x5645fbab51b0 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.687+0000 7fb6dffff640 1 -- 172.21.15.28:0/4272737300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f001e8e0 con 0x5645fbab51b0 2024-09-21T05:47:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.687+0000 7fb6eaffd640 1 -- 172.21.15.28:0/3870968676 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb6f00e7410 con 0x7fb6f4102a60 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.688+0000 7fb6fa744640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 0x7fb6e00b8b90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4111c70 tx=0x7fb6ec02a8e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.688+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 msgr2=0x5645fbe857d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00acf80 tx=0x7fb6f00ddd20 comp rx=0 tx=0).stop 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 shutdown_connections 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe85d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe857d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 --2- 172.21.15.28:0/4272737300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe85290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 >> 172.21.15.28:0/4272737300 conn(0x5645fbad2460 msgr2=0x5645fbe83180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 shutdown_connections 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.689+0000 7fb70271b740 1 -- 172.21.15.28:0/4272737300 wait complete. 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.690+0000 7fb70271b740 1 -- start start 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.690+0000 7fb70271b740 1 Processor -- start 2024-09-21T05:47:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe90e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe9cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645fbc4c700 con 0x5645fbab51b0 2024-09-21T05:47:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5645fbd86160 con 0x5645fbabd390 2024-09-21T05:47:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5645fbe93a70 con 0x5645fbade070 2024-09-21T05:47:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb70271b740 10 client.0 _ll_register_callbacks cb 0x5645fbd9f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45022/0 (socket says 172.21.15.28:45022) 2024-09-21T05:47:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb6f9f43640 1 -- 172.21.15.28:0/2478137041 learned_addr learned my addr 172.21.15.28:0/2478137041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.691+0000 7fb6f9742640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe90e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 -- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 msgr2=0x5645fbe9cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe9cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 -- 172.21.15.28:0/2478137041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 msgr2=0x5645fbe90910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 -- 172.21.15.28:0/2478137041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645fbdb3c90 con 0x5645fbabd390 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6f9742640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe90e50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6f406aaa0 tx=0x7fb6e40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=220455208cd0e316 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.692+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4012860 con 0x5645fbabd390 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb6e4012a00 con 0x5645fbabd390 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6f00acd90 con 0x5645fbabd390 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6e4012c00 con 0x5645fbabd390 2024-09-21T05:47:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb70271b740 10 client.25370 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5645fbe9f780 con 0x5645fbabd390 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f00acd90 con 0x5645fbabd390 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.693+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb6e4021020 con 0x5645fbabd390 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 10 client.25370 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 10 client.25370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 20 client.25370 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fba9c9f0 0x5645fbea3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 4 client.25370 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645fbdfddf0 con 0x5645fba9c9f0 2024-09-21T05:47:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fba9c9f0 0x5645fbea3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.694+0000 7fb6dd7fa640 10 client.25370 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.695+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fba9c9f0 0x5645fbea3230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf101f847aed4c6 server_cookie=512a8735397c8e2c in_seq=0 out_seq=0 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.696+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/2478137041 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5645fbdfddf0 con 0x5645fba9c9f0 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.696+0000 7fb6dd7fa640 10 client.25370 handle_command_reply: tid=0 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.696+0000 7fb70271b740 1 client.25370 shutdown 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.696+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 msgr2=0x5645fbe90e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.696+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe90e50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6f406aaa0 tx=0x7fb6e40117b0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 shutdown_connections 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fba9c9f0 0x5645fbea3230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbade070 0x5645fbe9cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbabd390 0x5645fbe90e50 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6f406aaa0 tx=0x7fb6e40117b0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 --2- 172.21.15.28:0/2478137041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbab51b0 0x5645fbe90910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb6dd7fa640 0 client.25370 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.697+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 >> 172.21.15.28:0/2478137041 conn(0x5645fbad2460 msgr2=0x5645fbe862c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.698+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 shutdown_connections 2024-09-21T05:47:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.698+0000 7fb70271b740 1 -- 172.21.15.28:0/2478137041 wait complete. 2024-09-21T05:47:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.699+0000 7fb70271b740 20 client.25370 trim_cache size 0 max 16384 2024-09-21T05:47:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.725+0000 7fb70271b740 1 Processor -- start 2024-09-21T05:47:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.725+0000 7fb70271b740 1 -- start start 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe9cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbade070 0x5645fbe82730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb6f9742640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbade070 0x5645fbe82730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b5e00 con 0x5645fbade070 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b7940 con 0x5645fba71da0 2024-09-21T05:47:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb70271b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b64d0 con 0x5645fbab51b0 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb6fa744640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb6fa744640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60850/0 (socket says 172.21.15.28:60850) 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb6fa744640 1 -- 172.21.15.28:0/2002380635 learned_addr learned my addr 172.21.15.28:0/2002380635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.726+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe9cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 -- 172.21.15.28:0/2002380635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 msgr2=0x5645fbe9cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe9cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 -- 172.21.15.28:0/2002380635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbade070 msgr2=0x5645fbe82730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbade070 0x5645fbe82730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 -- 172.21.15.28:0/2002380635 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645fbe7d460 con 0x5645fba71da0 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe9cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6fa744640 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec035000 tx=0x7fb6ec009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2737066d7e328fad server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6df7fe640 1 -- 172.21.15.28:0/2002380635 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec04f070 con 0x5645fba71da0 2024-09-21T05:47:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6df7fe640 1 -- 172.21.15.28:0/2002380635 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6ec036040 con 0x5645fba71da0 2024-09-21T05:47:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.727+0000 7fb6df7fe640 1 -- 172.21.15.28:0/2002380635 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec044070 con 0x5645fba71da0 2024-09-21T05:47:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.728+0000 7fb70271b740 1 -- 172.21.15.28:0/2002380635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 msgr2=0x5645fbe82c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.728+0000 7fb70271b740 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec035000 tx=0x7fb6ec009580 comp rx=0 tx=0).stop 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 -- 172.21.15.28:0/2002380635 shutdown_connections 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fba71da0 0x5645fbe82c70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fbade070 0x5645fbe82730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 --2- 172.21.15.28:0/2002380635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe9cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 -- 172.21.15.28:0/2002380635 >> 172.21.15.28:0/2002380635 conn(0x5645fbad2460 msgr2=0x5645fbe8f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 -- 172.21.15.28:0/2002380635 shutdown_connections 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 -- 172.21.15.28:0/2002380635 wait complete. 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 -- start start 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.729+0000 7fb70271b740 1 Processor -- start 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.730+0000 7fb70271b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.730+0000 7fb70271b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45038/0 (socket says 172.21.15.28:45038) 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3290353818 learned_addr learned my addr 172.21.15.28:0/3290353818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbade070 0x5645fbf0bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b7270 con 0x5645fba71da0 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b6ba0 con 0x5645fbade070 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f41b3de0 con 0x5645fbab51b0 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9742640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6fa744640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbade070 0x5645fbf0bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 msgr2=0x5645fbe949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe949d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3290353818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbade070 msgr2=0x5645fbf0bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbade070 0x5645fbf0bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.731+0000 7fb6f9f43640 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645fbf0c160 con 0x5645fba71da0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb6f9f43640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00acf80 tx=0x7fb6f0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c82ee64fdd86028 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb6dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb70271b740 10 client.0 _ll_register_callbacks cb 0x7fb6e4009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6ec036040 con 0x5645fba71da0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/3290353818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f004c910 con 0x5645fba71da0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/3290353818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb6f00eac50 con 0x5645fba71da0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.732+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/3290353818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6f0013d50 con 0x5645fba71da0 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 10 client.15753 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5645fbf0d1b0 con 0x5645fba71da0 2024-09-21T05:47:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/3290353818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb6f0082ce0 con 0x5645fba71da0 2024-09-21T05:47:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 10 client.15753 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 20 client.15753 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fbf0d940 0x5645fbf0fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 4 client.15753 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.733+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb6ec036040 con 0x5645fbf0d940 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.734+0000 7fb6f9742640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fbf0d940 0x5645fbf0fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.734+0000 7fb6dd7fa640 10 client.15753 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.734+0000 7fb6f9742640 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fbf0d940 0x5645fbf0fdb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99ab783b2a199cdf server_cookie=658aef6d72afee8e in_seq=0 out_seq=0 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.734+0000 7fb6dd7fa640 1 -- 172.21.15.28:0/3290353818 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb6ec036040 con 0x5645fbf0d940 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.734+0000 7fb6dd7fa640 10 client.15753 handle_command_reply: tid=0 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 client.15753 shutdown 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 msgr2=0x5645fbe94490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00acf80 tx=0x7fb6f0000bc0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 shutdown_connections 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5645fbf0d940 0x5645fbf0fdb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645fbade070 0x5645fbf0bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5645fbab51b0 0x5645fbe949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 --2- 172.21.15.28:0/3290353818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645fba71da0 0x5645fbe94490 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 >> 172.21.15.28:0/3290353818 conn(0x5645fbad2460 msgr2=0x5645fbe8ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb6dd7fa640 0 client.15753 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 shutdown_connections 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 1 -- 172.21.15.28:0/3290353818 wait complete. 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.735+0000 7fb70271b740 20 client.15753 trim_cache size 0 max 16384 2024-09-21T05:47:22.742 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 msgr2=0x7fb6e00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 0x7fb6e00b8b90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4111c70 tx=0x7fb6ec02a8e0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 msgr2=0x7fb6f41115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000d950 tx=0x7fb6f000de10 comp rx=0 tx=0).stop 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 shutdown_connections 2024-09-21T05:47:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6f4108cf0 0x7fb6f41100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6e00b6680 0x7fb6e00b8b90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6f41083a0 0x7fb6f410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 --2- 172.21.15.28:0/3870968676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6f4102a60 0x7fb6f41115c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.736+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 >> 172.21.15.28:0/3870968676 conn(0x7fb6f4079730 msgr2=0x7fb6f40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.737+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 shutdown_connections 2024-09-21T05:47:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:22.737+0000 7fb6fc1a7640 1 -- 172.21.15.28:0/3870968676 wait complete. 2024-09-21T05:47:24.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:23 smithi129 ceph-mon[34885]: pgmap v336: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T05:47:24.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:23 smithi181 ceph-mon[34819]: pgmap v336: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T05:47:24.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:23 smithi028 ceph-mon[31227]: pgmap v336: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-21T05:47:24.539 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:47:24.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:47:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 -- 172.21.15.28:0/4291924328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c4105300 msgr2=0x7fe4c410d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/4291924328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c4105300 0x7fe4c410d850 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009900 tx=0x7fe4b402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 -- 172.21.15.28:0/4291924328 shutdown_connections 2024-09-21T05:47:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/4291924328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c410de10 0x7fe4c4110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/4291924328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c4105300 0x7fe4c410d850 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/4291924328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 -- 172.21.15.28:0/4291924328 >> 172.21.15.28:0/4291924328 conn(0x7fe4c4079730 msgr2=0x7fe4c4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 -- 172.21.15.28:0/4291924328 shutdown_connections 2024-09-21T05:47:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.526+0000 7fe4cb35d640 1 -- 172.21.15.28:0/4291924328 wait complete. 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.527+0000 7fe4cb35d640 1 Processor -- start 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.527+0000 7fe4cb35d640 1 -- start start 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c4105300 0x7fe4c41026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4113ac0 con 0x7fe4c410de10 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4113940 con 0x7fe4c41049b0 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4cb35d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4113c40 con 0x7fe4c4105300 2024-09-21T05:47:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34618/0 (socket says 172.21.15.28:34618) 2024-09-21T05:47:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4c98fa640 1 -- 172.21.15.28:0/2922857607 learned_addr learned my addr 172.21.15.28:0/2922857607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.528+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c4105300 0x7fe4c41026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 -- 172.21.15.28:0/2922857607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c4105300 msgr2=0x7fe4c41026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c4105300 0x7fe4c41026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 -- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 msgr2=0x7fe4c4102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 -- 172.21.15.28:0/2922857607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c4103450 con 0x7fe4c41049b0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e9c0 tx=0x7fe4c000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4ba7fc640 1 -- 172.21.15.28:0/2922857607 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000ccc0 con 0x7fe4c41049b0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c41acdf0 con 0x7fe4c41049b0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4ba7fc640 1 -- 172.21.15.28:0/2922857607 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c00035f0 con 0x7fe4c41049b0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.529+0000 7fe4ba7fc640 1 -- 172.21.15.28:0/2922857607 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000ce60 con 0x7fe4c41049b0 2024-09-21T05:47:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.530+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c41ad1f0 con 0x7fe4c41049b0 2024-09-21T05:47:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.533+0000 7fe4d18d1740 1 Processor -- start 2024-09-21T05:47:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.533+0000 7fe4d18d1740 1 -- start start 2024-09-21T05:47:25.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef21fb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 0x558ef21fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ef2171e60 con 0x558ef1e54350 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ef2161290 con 0x558ef1e33670 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558ef21650a0 con 0x558ef1e2b490 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34632/0 (socket says 172.21.15.28:34632) 2024-09-21T05:47:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4c98fa640 1 -- 172.21.15.28:0/3516576350 learned_addr learned my addr 172.21.15.28:0/3516576350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.534+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 0x558ef21fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 -- 172.21.15.28:0/3516576350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 msgr2=0x558ef21fb970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef21fb970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 -- 172.21.15.28:0/3516576350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 msgr2=0x558ef21fbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 0x558ef21fbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 -- 172.21.15.28:0/3516576350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef1ebf930 con 0x558ef1e54350 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 0x558ef21fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e800 tx=0x7fe4bc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28e35c342a20de4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4af7fe640 1 -- 172.21.15.28:0/3516576350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc004d20 con 0x558ef1e54350 2024-09-21T05:47:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.535+0000 7fe4af7fe640 1 -- 172.21.15.28:0/3516576350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4bc005020 con 0x558ef1e54350 2024-09-21T05:47:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.536+0000 7fe4ba7fc640 1 -- 172.21.15.28:0/2922857607 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe4c0010630 con 0x7fe4c41049b0 2024-09-21T05:47:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.536+0000 7fe4af7fe640 1 -- 172.21.15.28:0/3516576350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc01a600 con 0x558ef1e54350 2024-09-21T05:47:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.536+0000 7fe4d18d1740 1 -- 172.21.15.28:0/3516576350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 msgr2=0x558ef21fb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.536+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e800 tx=0x7fe4bc00ecc0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 -- 172.21.15.28:0/3516576350 shutdown_connections 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e33670 0x558ef21fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef21fb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/3516576350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e54350 0x558ef21fb430 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 -- 172.21.15.28:0/3516576350 >> 172.21.15.28:0/3516576350 conn(0x558ef1e48740 msgr2=0x558ef21f9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 -- 172.21.15.28:0/3516576350 shutdown_connections 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4d18d1740 1 -- 172.21.15.28:0/3516576350 wait complete. 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.537+0000 7fe4ba7fc640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 0x7fe4b00b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.538+0000 7fe4d18d1740 1 -- start start 2024-09-21T05:47:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.538+0000 7fe4ba7fc640 1 -- 172.21.15.28:0/2922857607 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe4c00e7bd0 con 0x7fe4c41049b0 2024-09-21T05:47:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.538+0000 7fe4d18d1740 1 Processor -- start 2024-09-21T05:47:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.538+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 0x7fe4b00b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.539+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.539+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef2207440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.539+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.539+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ef1fc2890 con 0x558ef1e33670 2024-09-21T05:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ef20fbd40 con 0x558ef1e2b490 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558ef2209cb0 con 0x558ef1e54350 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c90f9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:48542/0 (socket says 172.21.15.28:48542) 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c98fa640 1 -- 172.21.15.28:0/4075752312 learned_addr learned my addr 172.21.15.28:0/4075752312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c90f9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59782/0 (socket says 172.21.15.28:59782) 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 0x7fe4b00b8ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.540+0000 7fe4d18d1740 10 client.0 _ll_register_callbacks cb 0x558ef215fc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef2207440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 -- 172.21.15.28:0/4075752312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 msgr2=0x558ef2206f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 -- 172.21.15.28:0/4075752312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 msgr2=0x558ef2207440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef2207440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 -- 172.21.15.28:0/4075752312 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef2213530 con 0x558ef1e54350 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005f40 tx=0x7fe4bc004f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a24d9abea0b52c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4acff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.541+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc02a070 con 0x558ef1e54350 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe4bc003970 con 0x558ef1e54350 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4bc005020 con 0x558ef1e54350 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4bc033050 con 0x558ef1e54350 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4d18d1740 10 client.25435 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ef2215dd0 con 0x558ef1e54350 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.542+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4bc005680 con 0x558ef1e54350 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.543+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe4bc02d020 con 0x558ef1e54350 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 10 client.25435 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 10 client.25435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 20 client.25435 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1e12cd0 0x558ef22191e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 4 client.25435 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe4bc005020 con 0x558ef1e12cd0 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1e12cd0 0x558ef22191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1e12cd0 0x558ef22191e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=440b8492102c5499 server_cookie=216e1f4483d5bce4 in_seq=0 out_seq=0 2024-09-21T05:47:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.544+0000 7fe4acff9640 10 client.25435 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.545+0000 7fe4acff9640 1 -- 172.21.15.28:0/4075752312 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fe4bc005020 con 0x558ef1e12cd0 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.545+0000 7fe4acff9640 10 client.25435 handle_command_reply: tid=0 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.545+0000 7fe4d18d1740 1 client.25435 shutdown 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 msgr2=0x558ef2212ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005f40 tx=0x7fe4bc004f60 comp rx=0 tx=0).stop 2024-09-21T05:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 shutdown_connections 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1e12cd0 0x558ef22191e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4acff9640 0 client.25435 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e54350 0x558ef2212ff0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef2207440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/4075752312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e2b490 0x558ef2206f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 >> 172.21.15.28:0/4075752312 conn(0x558ef1e48740 msgr2=0x558ef21fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 shutdown_connections 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 1 -- 172.21.15.28:0/4075752312 wait complete. 2024-09-21T05:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.546+0000 7fe4d18d1740 20 client.25435 trim_cache size 0 max 16384 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.581+0000 7fe4d18d1740 1 Processor -- start 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 -- start start 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef2212ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef21fce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b5c50 con 0x558ef1e33670 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b7790 con 0x558ef1e54350 2024-09-21T05:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4d18d1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b6320 con 0x558ef1e2b490 2024-09-21T05:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4c88f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef21fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4c98fa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59792/0 (socket says 172.21.15.28:59792) 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.582+0000 7fe4c98fa640 1 -- 172.21.15.28:0/1350972579 learned_addr learned my addr 172.21.15.28:0/1350972579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 -- 172.21.15.28:0/1350972579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 msgr2=0x558ef2212ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef2212ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 -- 172.21.15.28:0/1350972579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 msgr2=0x558ef21fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef21fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 -- 172.21.15.28:0/1350972579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef2207fc0 con 0x558ef1e54350 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef21fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc002230 tx=0x7fe4bc005a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3f91f55a10f28b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4aeffd640 1 -- 172.21.15.28:0/1350972579 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc02a040 con 0x558ef1e54350 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4aeffd640 1 -- 172.21.15.28:0/1350972579 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4bc003e20 con 0x558ef1e54350 2024-09-21T05:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.583+0000 7fe4aeffd640 1 -- 172.21.15.28:0/1350972579 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0041c0 con 0x558ef1e54350 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.584+0000 7fe4d18d1740 1 -- 172.21.15.28:0/1350972579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 msgr2=0x558ef22065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.584+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc002230 tx=0x7fe4bc005a70 comp rx=0 tx=0).stop 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 -- 172.21.15.28:0/1350972579 shutdown_connections 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22065d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef21fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/1350972579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef2212ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 -- 172.21.15.28:0/1350972579 >> 172.21.15.28:0/1350972579 conn(0x558ef1e48740 msgr2=0x558ef2208220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 -- 172.21.15.28:0/1350972579 shutdown_connections 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 -- 172.21.15.28:0/1350972579 wait complete. 2024-09-21T05:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.585+0000 7fe4d18d1740 1 -- start start 2024-09-21T05:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4d18d1740 1 Processor -- start 2024-09-21T05:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef220b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4d18d1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4c90f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef220b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4c88f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4c90f9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef220b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:48558/0 (socket says 172.21.15.28:48558) 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4c88f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34660/0 (socket says 172.21.15.28:34660) 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4c90f9640 1 -- 172.21.15.28:0/589875577 learned_addr learned my addr 172.21.15.28:0/589875577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.586+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22829e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b70c0 con 0x558ef1e33670 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b69f0 con 0x558ef1e54350 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41b3c30 con 0x558ef1e2b490 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c98fa640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22829e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c88f8640 1 -- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 msgr2=0x558ef220b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef220b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c88f8640 1 -- 172.21.15.28:0/589875577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 msgr2=0x558ef22829e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22829e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4c88f8640 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef220bf80 con 0x558ef1e33670 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4d18d1740 10 client.0 _ll_register_callbacks cb 0x7fe4bc00e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.587+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4bc003e20 con 0x558ef1e33670 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4c88f8640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009230 tx=0x7fe4b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592b8c020f3ce0a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4acff9640 10 client.15771 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4acff9640 1 -- 172.21.15.28:0/589875577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4035050 con 0x558ef1e33670 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4acff9640 1 -- 172.21.15.28:0/589875577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe4b4005230 con 0x558ef1e33670 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4acff9640 1 -- 172.21.15.28:0/589875577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4b4005470 con 0x558ef1e33670 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.588+0000 7fe4d18d1740 10 client.15771 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=27}) -- 0x558ef2283c90 con 0x558ef1e33670 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4acff9640 1 -- 172.21.15.28:0/589875577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe4b40046c0 con 0x558ef1e33670 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 10 client.15771 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 20 client.15771 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1de8080 0x558ef2286460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 4 client.15771 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.589+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558ef2173db0 con 0x558ef1de8080 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.590+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1de8080 0x558ef2286460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.590+0000 7fe4acff9640 10 client.15771 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.590+0000 7fe4c90f9640 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1de8080 0x558ef2286460 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9243c573f5a3d11b server_cookie=d74c43ec42d11532 in_seq=0 out_seq=0 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4acff9640 1 -- 172.21.15.28:0/589875577 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558ef2173db0 con 0x558ef1de8080 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4acff9640 10 client.15771 handle_command_reply: tid=0 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 client.15771 shutdown 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 msgr2=0x558ef220b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009230 tx=0x7fe4b4009260 comp rx=0 tx=0).stop 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 shutdown_connections 2024-09-21T05:47:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558ef1de8080 0x558ef2286460 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.592+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef1e54350 0x558ef22829e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef1e33670 0x558ef220b730 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 1 --2- 172.21.15.28:0/589875577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef1e2b490 0x558ef220b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 >> 172.21.15.28:0/589875577 conn(0x558ef1e48740 msgr2=0x558ef21fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4acff9640 0 client.15771 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 shutdown_connections 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 1 -- 172.21.15.28:0/589875577 wait complete. 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.593+0000 7fe4d18d1740 20 client.15771 trim_cache size 0 max 16384 2024-09-21T05:47:25.596 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:47:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941", 2024-09-21T05:47:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:47:25.597 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 msgr2=0x7fe4b00b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 0x7fe4b00b8ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b4031400 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 msgr2=0x7fe4c4104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e9c0 tx=0x7fe4c000ee80 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 shutdown_connections 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe4b00b65c0 0x7fe4b00b8ad0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4c410de10 0x7fe4c4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c4105300 0x7fe4c41026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 --2- 172.21.15.28:0/2922857607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c41049b0 0x7fe4c4104100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 >> 172.21.15.28:0/2922857607 conn(0x7fe4c4079730 msgr2=0x7fe4c4101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 shutdown_connections 2024-09-21T05:47:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:25.594+0000 7fe4cb35d640 1 -- 172.21.15.28:0/2922857607 wait complete. 2024-09-21T05:47:26.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:25 smithi129 ceph-mon[34885]: pgmap v337: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-21T05:47:26.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:25 smithi028 ceph-mon[31227]: pgmap v337: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-21T05:47:26.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:47:26.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:25 smithi181 ceph-mon[34819]: pgmap v337: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-21T05:47:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:26 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:47:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:26 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:26 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:47:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:26 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:26 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:47:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.025+0000 7fc95b243640 1 -- 172.21.15.28:0/2267660336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9541049d0 msgr2=0x7fc954104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.025+0000 7fc95b243640 1 --2- 172.21.15.28:0/2267660336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9541049d0 0x7fc954104de0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc948009900 tx=0x7fc94802a410 comp rx=0 tx=0).stop 2024-09-21T05:47:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.025+0000 7fc95b243640 1 -- 172.21.15.28:0/2267660336 shutdown_connections 2024-09-21T05:47:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.025+0000 7fc95b243640 1 --2- 172.21.15.28:0/2267660336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc95410de30 0x7fc954110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.026+0000 7fc95b243640 1 --2- 172.21.15.28:0/2267660336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc954105320 0x7fc95410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.026+0000 7fc95b243640 1 --2- 172.21.15.28:0/2267660336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9541049d0 0x7fc954104de0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.026+0000 7fc95b243640 1 -- 172.21.15.28:0/2267660336 >> 172.21.15.28:0/2267660336 conn(0x7fc954079740 msgr2=0x7fc954079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.026+0000 7fc95b243640 1 -- 172.21.15.28:0/2267660336 shutdown_connections 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.026+0000 7fc95b243640 1 -- 172.21.15.28:0/2267660336 wait complete. 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 Processor -- start 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 -- start start 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9541049d0 0x7fc954198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410de30 0x7fc95419ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc954113a90 con 0x7fc954105320 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc954113910 con 0x7fc95410de30 2024-09-21T05:47:27.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc95b243640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc954113c10 con 0x7fc9541049d0 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.027+0000 7fc953fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34678/0 (socket says 172.21.15.28:34678) 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 -- 172.21.15.28:0/1074855857 learned_addr learned my addr 172.21.15.28:0/1074855857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc958fdf640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9541049d0 0x7fc954198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc9597e0640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410de30 0x7fc95419ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 -- 172.21.15.28:0/1074855857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9541049d0 msgr2=0x7fc954198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9541049d0 0x7fc954198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 -- 172.21.15.28:0/1074855857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410de30 msgr2=0x7fc95419ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410de30 0x7fc95419ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.028+0000 7fc953fff640 1 -- 172.21.15.28:0/1074855857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9541a0500 con 0x7fc954105320 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.029+0000 7fc953fff640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc94000d740 tx=0x7fc94000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.029+0000 7fc951ffb640 1 -- 172.21.15.28:0/1074855857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940017070 con 0x7fc954105320 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.029+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9541a07f0 con 0x7fc954105320 2024-09-21T05:47:27.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.029+0000 7fc951ffb640 1 -- 172.21.15.28:0/1074855857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc94000de50 con 0x7fc954105320 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.029+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541a0d80 con 0x7fc954105320 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.031+0000 7fc951ffb640 1 -- 172.21.15.28:0/1074855857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94000f620 con 0x7fc954105320 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.033+0000 7fc9617b7740 1 Processor -- start 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.033+0000 7fc9617b7740 1 -- start start 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.033+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 0x562d61ce5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61ce60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9617b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d61c5be50 con 0x562d6193e070 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9617b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562d61c4b280 con 0x562d619151b0 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9617b7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562d61c4f090 con 0x562d6191d390 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34688/0 (socket says 172.21.15.28:34688) 2024-09-21T05:47:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9597e0640 1 -- 172.21.15.28:0/2309530940 learned_addr learned my addr 172.21.15.28:0/2309530940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc951ffb640 1 -- 172.21.15.28:0/1074855857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc94001a070 con 0x7fc954105320 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 0x562d61ce5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 -- 172.21.15.28:0/2309530940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 msgr2=0x562d61ce60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61ce60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 -- 172.21.15.28:0/2309530940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 msgr2=0x562d61ce5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc958fdf640 1 -- 172.21.15.28:0/2309530940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d619a9650 con 0x562d619151b0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.034+0000 7fc9597e0640 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.035+0000 7fc958fdf640 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 0x562d61ce5b90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc948006010 tx=0x7fc948000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dabdfb38728eb11 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.035+0000 7fc93effd640 1 -- 172.21.15.28:0/2309530940 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948040070 con 0x562d619151b0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.035+0000 7fc93effd640 1 -- 172.21.15.28:0/2309530940 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc948004030 con 0x562d619151b0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.035+0000 7fc93effd640 1 -- 172.21.15.28:0/2309530940 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948035040 con 0x562d619151b0 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.035+0000 7fc951ffb640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 0x7fc9440b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc951ffb640 1 -- 172.21.15.28:0/1074855857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc9400e6ff0 con 0x7fc954105320 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9597e0640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 0x7fc9440b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 -- 172.21.15.28:0/2309530940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 msgr2=0x562d61ce5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9597e0640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 0x7fc9440b8c10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc94c011890 tx=0x7fc94c009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 0x562d61ce5b90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc948006010 tx=0x7fc948000bc0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 -- 172.21.15.28:0/2309530940 shutdown_connections 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61ce60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d619151b0 0x562d61ce5b90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 --2- 172.21.15.28:0/2309530940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61ce5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 -- 172.21.15.28:0/2309530940 >> 172.21.15.28:0/2309530940 conn(0x562d61932460 msgr2=0x562d61ce3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.036+0000 7fc9617b7740 1 -- 172.21.15.28:0/2309530940 shutdown_connections 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.037+0000 7fc9617b7740 1 -- 172.21.15.28:0/2309530940 wait complete. 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.037+0000 7fc9617b7740 1 -- start start 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.037+0000 7fc9617b7740 1 Processor -- start 2024-09-21T05:47:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61cf0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cf12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cfd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d61aac700 con 0x562d6193e070 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562d61be6110 con 0x562d6191d390 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562d61cf3dd0 con 0x562d619151b0 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9617b7740 10 client.0 _ll_register_callbacks cb 0x562d61bff990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cfd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cfd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34696/0 (socket says 172.21.15.28:34696) 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc9597e0640 1 -- 172.21.15.28:0/4280483329 learned_addr learned my addr 172.21.15.28:0/4280483329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61cf0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc953fff640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cf12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 -- 172.21.15.28:0/4280483329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 msgr2=0x562d61cf12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cf12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 -- 172.21.15.28:0/4280483329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 msgr2=0x562d61cfd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cfd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.038+0000 7fc958fdf640 1 -- 172.21.15.28:0/4280483329 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d61c13c40 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc958fdf640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61cf0db0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc94802a930 tx=0x7fc948032f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f78af1dda7f2543a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc933fff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948035070 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc948004030 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc948005280 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc948040040 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc9617b7740 10 client.25447 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562d61cffb70 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.039+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc948004030 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc948054030 con 0x562d619151b0 2024-09-21T05:47:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 10 client.25447 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 10 client.25447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 20 client.25447 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618fc9f0 0x562d61d02ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 4 client.25447 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d61c5dda0 con 0x562d618fc9f0 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc953fff640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618fc9f0 0x562d61d02ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc933fff640 10 client.25447 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.040+0000 7fc953fff640 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618fc9f0 0x562d61d02ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96360fb415ac0ba7 server_cookie=3339c7165f94c7d9 in_seq=0 out_seq=0 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.041+0000 7fc933fff640 1 -- 172.21.15.28:0/4280483329 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562d61c5dda0 con 0x562d618fc9f0 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.041+0000 7fc933fff640 10 client.25447 handle_command_reply: tid=0 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.041+0000 7fc9617b7740 1 client.25447 shutdown 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.041+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 msgr2=0x562d61cf0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.041+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61cf0db0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc94802a930 tx=0x7fc948032f80 comp rx=0 tx=0).stop 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 shutdown_connections 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618fc9f0 0x562d61d02ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cfd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc933fff640 0 client.25447 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cf12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 --2- 172.21.15.28:0/4280483329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61cf0db0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 >> 172.21.15.28:0/4280483329 conn(0x562d61932460 msgr2=0x562d61ce8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 shutdown_connections 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 1 -- 172.21.15.28:0/4280483329 wait complete. 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.042+0000 7fc9617b7740 20 client.25447 trim_cache size 0 max 16384 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 Processor -- start 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 -- start start 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61ce8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cedfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541be560 con 0x562d6193e070 2024-09-21T05:47:27.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541c00a0 con 0x562d6191d390 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.064+0000 7fc9617b7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541bec30 con 0x562d619151b0 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc9597e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34700/0 (socket says 172.21.15.28:34700) 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc9597e0640 1 -- 172.21.15.28:0/3586890748 learned_addr learned my addr 172.21.15.28:0/3586890748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cedfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc958fdf640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61ce8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 -- 172.21.15.28:0/3586890748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 msgr2=0x562d61ce8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61ce8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 -- 172.21.15.28:0/3586890748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 msgr2=0x562d61cee4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 -- 172.21.15.28:0/3586890748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d61cf3960 con 0x562d6191d390 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc9597e0640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc958fdf640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61ce8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:27.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.065+0000 7fc953fff640 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cedfa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9400e4740 tx=0x7fc9400e9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a4fa68ebd436a1b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.066+0000 7fc93e7fc640 1 -- 172.21.15.28:0/3586890748 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9400af050 con 0x562d6191d390 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.066+0000 7fc93e7fc640 1 -- 172.21.15.28:0/3586890748 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9400cee50 con 0x562d6191d390 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.066+0000 7fc93e7fc640 1 -- 172.21.15.28:0/3586890748 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940086e60 con 0x562d6191d390 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- 172.21.15.28:0/3586890748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 msgr2=0x562d61cedfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cedfa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9400e4740 tx=0x7fc9400e9040 comp rx=0 tx=0).stop 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- 172.21.15.28:0/3586890748 shutdown_connections 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d6193e070 0x562d61cee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6191d390 0x562d61cedfa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3586890748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d619151b0 0x562d61ce8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- 172.21.15.28:0/3586890748 >> 172.21.15.28:0/3586890748 conn(0x562d61932460 msgr2=0x562d61ce8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- 172.21.15.28:0/3586890748 shutdown_connections 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- 172.21.15.28:0/3586890748 wait complete. 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 -- start start 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.067+0000 7fc9617b7740 1 Processor -- start 2024-09-21T05:47:27.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61d04610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 0x562d61d04b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541bf9d0 con 0x562d619151b0 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541bf300 con 0x562d6193e070 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541bc540 con 0x562d6191d390 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc9617b7740 10 client.0 _ll_register_callbacks cb 0x7fc94c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc958fdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc958fdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34702/0 (socket says 172.21.15.28:34702) 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.068+0000 7fc958fdf640 1 -- 172.21.15.28:0/3111899964 learned_addr learned my addr 172.21.15.28:0/3111899964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc9597e0640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 0x562d61d04b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc953fff640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61d04610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 -- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 msgr2=0x562d61d04610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61d04610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 -- 172.21.15.28:0/3111899964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 msgr2=0x562d61d04b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 0x562d61d04b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 -- 172.21.15.28:0/3111899964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d61d6c850 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc9597e0640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 0x562d61d04b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc958fdf640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc94802a930 tx=0x7fc948005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4556d68b7ae4b48c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc933fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948035040 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.069+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc9480048c0 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9400cee50 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc9480051f0 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 10 client.15789 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562d61d6dac0 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc948005390 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc948064020 con 0x562d619151b0 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 10 client.15789 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 20 client.15789 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618d1da0 0x562d61d701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 4 client.15789 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.070+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fc9400cee50 con 0x562d618d1da0 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.071+0000 7fc953fff640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618d1da0 0x562d61d701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.071+0000 7fc933fff640 10 client.15789 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc953fff640 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618d1da0 0x562d61d701e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc74a9eaacd5e66 server_cookie=572fdc1eb17e752 in_seq=0 out_seq=0 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc933fff640 1 -- 172.21.15.28:0/3111899964 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+658 (crc 0 0 0) 0x7fc9400cee50 con 0x562d618d1da0 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc933fff640 10 client.15789 handle_command_reply: tid=0 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc9617b7740 1 client.15789 shutdown 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 msgr2=0x562d61d040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc94802a930 tx=0x7fc948005f00 comp rx=0 tx=0).stop 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 shutdown_connections 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.072+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562d618d1da0 0x562d61d701e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d6193e070 0x562d61d04b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562d6191d390 0x562d61d04610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 --2- 172.21.15.28:0/3111899964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d619151b0 0x562d61d040d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc933fff640 0 client.15789 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 >> 172.21.15.28:0/3111899964 conn(0x562d61932460 msgr2=0x562d61ce8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 shutdown_connections 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:47:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 1 -- 172.21.15.28:0/3111899964 wait complete. 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.073+0000 7fc9617b7740 20 client.15789 trim_cache size 0 max 16384 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 msgr2=0x7fc9440b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 0x7fc9440b8c10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc94c011890 tx=0x7fc94c009500 comp rx=0 tx=0).stop 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 msgr2=0x7fc9541994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc94000d740 tx=0x7fc94000dc00 comp rx=0 tx=0).stop 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 shutdown_connections 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410de30 0x7fc95419ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9440b6700 0x7fc9440b8c10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc954105320 0x7fc9541994a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 --2- 172.21.15.28:0/1074855857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9541049d0 0x7fc954198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 >> 172.21.15.28:0/1074855857 conn(0x7fc954079740 msgr2=0x7fc95410cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 shutdown_connections 2024-09-21T05:47:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:27.074+0000 7fc95b243640 1 -- 172.21.15.28:0/1074855857 wait complete. 2024-09-21T05:47:27.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:26 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:47:27.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:26 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:26 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:47:27.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:26 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:26 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:47:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:26 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:47:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:26 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:26 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:47:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:26 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:26 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:47:27.758 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:47:27.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:47:28.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:27 smithi028 ceph-mon[31227]: pgmap v338: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-21T05:47:28.074 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:28.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:27 smithi181 ceph-mon[34819]: pgmap v338: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-21T05:47:28.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:27 smithi129 ceph-mon[34885]: pgmap v338: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-21T05:47:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 -- 172.21.15.28:0/2966767947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 msgr2=0x7fab9c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 --2- 172.21.15.28:0/2966767947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c110300 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fab94009ca0 tx=0x7fab9402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 -- 172.21.15.28:0/2966767947 shutdown_connections 2024-09-21T05:47:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 --2- 172.21.15.28:0/2966767947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c110300 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 --2- 172.21.15.28:0/2966767947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c105300 0x7fab9c10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 --2- 172.21.15.28:0/2966767947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c1049b0 0x7fab9c104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 -- 172.21.15.28:0/2966767947 >> 172.21.15.28:0/2966767947 conn(0x7fab9c079730 msgr2=0x7fab9c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 -- 172.21.15.28:0/2966767947 shutdown_connections 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.607+0000 7faba256c640 1 -- 172.21.15.28:0/2966767947 wait complete. 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 Processor -- start 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 -- start start 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 0x7fab9c109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c105300 0x7fab9c105910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c113a70 con 0x7fab9c10de10 2024-09-21T05:47:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c1138f0 con 0x7fab9c1049b0 2024-09-21T05:47:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.608+0000 7faba256c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c113bf0 con 0x7fab9c105300 2024-09-21T05:47:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34716/0 (socket says 172.21.15.28:34716) 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 -- 172.21.15.28:0/962955232 learned_addr learned my addr 172.21.15.28:0/962955232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7fab9b7fe640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c105300 0x7fab9c105910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7fab9bfff640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 0x7fab9c109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 -- 172.21.15.28:0/962955232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c105300 msgr2=0x7fab9c105910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c105300 0x7fab9c105910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 -- 172.21.15.28:0/962955232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 msgr2=0x7fab9c109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 0x7fab9c109630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7faba0b09640 1 -- 172.21.15.28:0/962955232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c1acbb0 con 0x7fab9c10de10 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.609+0000 7fab9bfff640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 0x7fab9c109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7faba0b09640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fab94005cc0 tx=0x7fab94031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94003b70 con 0x7fab9c10de10 2024-09-21T05:47:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c1acde0 con 0x7fab9c10de10 2024-09-21T05:47:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab940318f0 con 0x7fab9c10de10 2024-09-21T05:47:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9402f950 con 0x7fab9c10de10 2024-09-21T05:47:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.610+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c1ad290 con 0x7fab9c10de10 2024-09-21T05:47:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.611+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c104dc0 con 0x7fab9c10de10 2024-09-21T05:47:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.613+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fab94004b40 con 0x7fab9c10de10 2024-09-21T05:47:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.614+0000 7fab997fa640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 0x7fab840b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.614+0000 7fab9bfff640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 0x7fab840b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.614+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fab94108a00 con 0x7fab9c10de10 2024-09-21T05:47:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.615+0000 7fab9bfff640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 0x7fab840b8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab9c109380 tx=0x7fab8c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.620+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fab940cef70 con 0x7fab9c10de10 2024-09-21T05:47:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.986+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab9c1ad6d0 con 0x7fab9c10de10 2024-09-21T05:47:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.987+0000 7fab997fa640 1 -- 172.21.15.28:0/962955232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7fab940d3350 con 0x7fab9c10de10 2024-09-21T05:47:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:47:28.989 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:47:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:47:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.989+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 msgr2=0x7fab840b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.989+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 0x7fab840b8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab9c109380 tx=0x7fab8c005f90 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.989+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 msgr2=0x7fab9c106030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.989+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fab94005cc0 tx=0x7fab94031700 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 shutdown_connections 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fab840b6700 0x7fab840b8c10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9c10de10 0x7fab9c106030 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab9c105300 0x7fab9c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 --2- 172.21.15.28:0/962955232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab9c1049b0 0x7fab9c109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 >> 172.21.15.28:0/962955232 conn(0x7fab9c079730 msgr2=0x7fab9c1036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 shutdown_connections 2024-09-21T05:47:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:28.990+0000 7faba256c640 1 -- 172.21.15.28:0/962955232 wait complete. 2024-09-21T05:47:29.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:47:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:30.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:29 smithi028 ceph-mon[31227]: pgmap v339: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-21T05:47:30.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/962955232' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:30.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:29 smithi181 ceph-mon[34819]: pgmap v339: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-21T05:47:30.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/962955232' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:30.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:29 smithi129 ceph-mon[34885]: pgmap v339: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-21T05:47:30.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/962955232' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 -- 172.21.15.28:0/2382653982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594102070 msgr2=0x7fb594102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 --2- 172.21.15.28:0/2382653982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594102070 0x7fb594102480 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb584009900 tx=0x7fb58402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 -- 172.21.15.28:0/2382653982 shutdown_connections 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 --2- 172.21.15.28:0/2382653982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb594106c10 0x7fb5941106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 --2- 172.21.15.28:0/2382653982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5941062c0 0x7fb5941066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 --2- 172.21.15.28:0/2382653982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594102070 0x7fb594102480 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.479+0000 7fb59943c640 1 -- 172.21.15.28:0/2382653982 >> 172.21.15.28:0/2382653982 conn(0x7fb594079e10 msgr2=0x7fb59407a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.480+0000 7fb59943c640 1 -- 172.21.15.28:0/2382653982 shutdown_connections 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.480+0000 7fb59943c640 1 -- 172.21.15.28:0/2382653982 wait complete. 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.480+0000 7fb59943c640 1 Processor -- start 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 -- start start 2024-09-21T05:47:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 0x7fb59419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5941062c0 0x7fb59419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb594113ae0 con 0x7fb594106c10 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb594113960 con 0x7fb5941062c0 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb59943c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb594113c60 con 0x7fb594102070 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34738/0 (socket says 172.21.15.28:34738) 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb5937fe640 1 -- 172.21.15.28:0/509336429 learned_addr learned my addr 172.21.15.28:0/509336429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb5927fc640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5941062c0 0x7fb59419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.481+0000 7fb592ffd640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 0x7fb59419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 -- 172.21.15.28:0/509336429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 msgr2=0x7fb59419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 0x7fb59419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 -- 172.21.15.28:0/509336429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5941062c0 msgr2=0x7fb59419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5941062c0 0x7fb59419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 -- 172.21.15.28:0/509336429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5941a04e0 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb592ffd640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 0x7fb59419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb5937fe640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009fd0 tx=0x7fb58c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb57bfff640 1 -- 172.21.15.28:0/509336429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c017070 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5941a07d0 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5941a6f30 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.482+0000 7fb57bfff640 1 -- 172.21.15.28:0/509336429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb58c003e20 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.483+0000 7fb57bfff640 1 -- 172.21.15.28:0/509336429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c010460 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.486+0000 7fb59f9b0740 1 Processor -- start 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.486+0000 7fb57bfff640 1 -- 172.21.15.28:0/509336429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb58c010720 con 0x7fb594106c10 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.486+0000 7fb59f9b0740 1 -- start start 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e2f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e2f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a123da5ea0 con 0x55a123a67390 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a123d952d0 con 0x55a123a5f1b0 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a123d990e0 con 0x55a123a88070 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34752/0 (socket says 172.21.15.28:34752) 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 -- 172.21.15.28:0/2025042734 learned_addr learned my addr 172.21.15.28:0/2025042734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e2f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e2f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 -- 172.21.15.28:0/2025042734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 msgr2=0x55a123e2f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e2f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 -- 172.21.15.28:0/2025042734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 msgr2=0x55a123e2f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e2f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb5927fc640 1 -- 172.21.15.28:0/2025042734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a123af3650 con 0x55a123a67390 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e2f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.487+0000 7fb57bfff640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 0x7fb57c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb58800bf80 tx=0x7fb588005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92315fa286382df9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb592ffd640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 0x7fb57c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb57bfff640 1 -- 172.21.15.28:0/509336429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb58c0e7120 con 0x7fb594106c10 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb5797fa640 1 -- 172.21.15.28:0/2025042734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588004230 con 0x55a123a67390 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb5797fa640 1 -- 172.21.15.28:0/2025042734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5880043d0 con 0x55a123a67390 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.488+0000 7fb5797fa640 1 -- 172.21.15.28:0/2025042734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588010810 con 0x55a123a67390 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2025042734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 msgr2=0x55a123e2fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb592ffd640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 0x7fb57c0b8c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb584000c00 tx=0x7fb584005ee0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb58800bf80 tx=0x7fb588005c90 comp rx=0 tx=0).stop 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2025042734 shutdown_connections 2024-09-21T05:47:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e2fd40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e2f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2025042734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e2f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2025042734 >> 172.21.15.28:0/2025042734 conn(0x55a123a7c460 msgr2=0x55a123e2d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2025042734 shutdown_connections 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.489+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2025042734 wait complete. 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.490+0000 7fb59f9b0740 1 -- start start 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.490+0000 7fb59f9b0740 1 Processor -- start 2024-09-21T05:47:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.490+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e3aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.490+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.490+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e46df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a123bf6700 con 0x55a123a67390 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a123d30160 con 0x55a123a5f1b0 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a123e3daa0 con 0x55a123a88070 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34766/0 (socket says 172.21.15.28:34766) 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 -- 172.21.15.28:0/2038282040 learned_addr learned my addr 172.21.15.28:0/2038282040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e3aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb59f9b0740 10 client.0 _ll_register_callbacks cb 0x55a123d499e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5937fe640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e46df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 -- 172.21.15.28:0/2038282040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 msgr2=0x55a123e46df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e46df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 -- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 msgr2=0x55a123e3aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e3aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 -- 172.21.15.28:0/2038282040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a123d5dc90 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb588001eb0 tx=0x7fb588004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1680b68a5ed9ff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.491+0000 7fb576ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5880054d0 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb5880035f0 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5880043d0 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb5880037f0 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb59f9b0740 10 client.15807 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a123e49870 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.492+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5880043d0 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb58802e070 con 0x55a123a67390 2024-09-21T05:47:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 10 client.15807 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 10 client.15807 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 20 client.15807 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a469f0 0x55a123e4cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 4 client.15807 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.493+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a123da7df0 con 0x55a123a469f0 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.494+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a469f0 0x55a123e4cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.494+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a469f0 0x55a123e4cbd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a0b56aecca2a9fe6 server_cookie=e6bbd85cd2079f0c in_seq=0 out_seq=0 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.494+0000 7fb576ff5640 10 client.15807 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb576ff5640 1 -- 172.21.15.28:0/2038282040 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a123da7df0 con 0x55a123a469f0 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb576ff5640 10 client.15807 handle_command_reply: tid=0 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 client.15807 shutdown 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 msgr2=0x55a123e3af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb588001eb0 tx=0x7fb588004230 comp rx=0 tx=0).stop 2024-09-21T05:47:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 shutdown_connections 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a469f0 0x55a123e4cbd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e46df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a67390 0x55a123e3af90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2038282040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a5f1b0 0x55a123e3aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 >> 172.21.15.28:0/2038282040 conn(0x55a123a7c460 msgr2=0x55a123e32450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb576ff5640 0 client.15807 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 shutdown_connections 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.495+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2038282040 wait complete. 2024-09-21T05:47:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.496+0000 7fb59f9b0740 20 client.15807 trim_cache size 0 max 16384 2024-09-21T05:47:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.522+0000 7fb59f9b0740 1 Processor -- start 2024-09-21T05:47:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.522+0000 7fb59f9b0740 1 -- start start 2024-09-21T05:47:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.522+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.522+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e32900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.522+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e37c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb58000e610 con 0x55a123a5f1b0 2024-09-21T05:47:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb580010150 con 0x55a123a67390 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb58000ece0 con 0x55a123a88070 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34768/0 (socket says 172.21.15.28:34768) 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 -- 172.21.15.28:0/2152844593 learned_addr learned my addr 172.21.15.28:0/2152844593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb5937fe640 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e37c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb5927fc640 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e32900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 -- 172.21.15.28:0/2152844593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 msgr2=0x55a123e37c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e37c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 -- 172.21.15.28:0/2152844593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 msgr2=0x55a123e32900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e32900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 -- 172.21.15.28:0/2152844593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a123e47590 con 0x55a123a5f1b0 2024-09-21T05:47:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.523+0000 7fb592ffd640 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb584009900 tx=0x7fb58403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50058397f3f02a9e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.524+0000 7fb578ff9640 1 -- 172.21.15.28:0/2152844593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584035050 con 0x55a123a5f1b0 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.524+0000 7fb578ff9640 1 -- 172.21.15.28:0/2152844593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb584054070 con 0x55a123a5f1b0 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.524+0000 7fb578ff9640 1 -- 172.21.15.28:0/2152844593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584004920 con 0x55a123a5f1b0 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2152844593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 msgr2=0x55a123e323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb584009900 tx=0x7fb58403d000 comp rx=0 tx=0).stop 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2152844593 shutdown_connections 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a88070 0x55a123e37c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e32900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/2152844593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a5f1b0 0x55a123e323c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2152844593 >> 172.21.15.28:0/2152844593 conn(0x55a123a7c460 msgr2=0x55a123e392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2152844593 shutdown_connections 2024-09-21T05:47:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- 172.21.15.28:0/2152844593 wait complete. 2024-09-21T05:47:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.525+0000 7fb59f9b0740 1 -- start start 2024-09-21T05:47:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 Processor -- start 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a5f1b0 0x55a123e3ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e3f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb58000fa80 con 0x55a123a88070 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb58000f3b0 con 0x55a123a67390 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb58000c6a0 con 0x55a123a5f1b0 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb59f9b0740 10 client.0 _ll_register_callbacks cb 0x7fb5880124a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb5937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34774/0 (socket says 172.21.15.28:34774) 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb5937fe640 1 -- 172.21.15.28:0/3856812437 learned_addr learned my addr 172.21.15.28:0/3856812437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb592ffd640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a5f1b0 0x55a123e3ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.526+0000 7fb5927fc640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e3f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 -- 172.21.15.28:0/3856812437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a5f1b0 msgr2=0x55a123e3ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a5f1b0 0x55a123e3ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 -- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 msgr2=0x55a123e3f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e3f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 -- 172.21.15.28:0/3856812437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a123eb63d0 con 0x55a123a88070 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5927fc640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e3f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb5937fe640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0acc90 tx=0x7fb58c0ac830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccc79c0b56d0bf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.527+0000 7fb576ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c020040 con 0x55a123a88070 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb58c0ac3f0 con 0x55a123a88070 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb584054070 con 0x55a123a88070 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb58c0b1a70 con 0x55a123a88070 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb59f9b0740 10 client.15813 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a123ebfe90 con 0x55a123a88070 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb58c0b69b0 con 0x55a123a88070 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb58c0abc60 con 0x55a123a88070 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.528+0000 7fb59f9b0740 10 client.15813 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb59f9b0740 10 client.15813 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb59f9b0740 20 client.15813 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a1bda0 0x55a123ec26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb59f9b0740 4 client.15813 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb584054070 con 0x55a123a1bda0 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb592ffd640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a1bda0 0x55a123ec26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb576ff5640 10 client.15813 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.529+0000 7fb592ffd640 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a1bda0 0x55a123ec26a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18a4fc2e8cbd0bdf server_cookie=a7023d7a6c1e2d69 in_seq=0 out_seq=0 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb576ff5640 1 -- 172.21.15.28:0/3856812437 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb584054070 con 0x55a123a1bda0 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb576ff5640 10 client.15813 handle_command_reply: tid=0 2024-09-21T05:47:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 client.15813 shutdown 2024-09-21T05:47:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 msgr2=0x55a123e3f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.536 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0acc90 tx=0x7fb58c0ac830 comp rx=0 tx=0).stop 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 shutdown_connections 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55a123a1bda0 0x55a123ec26a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a123a88070 0x55a123e3f7a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a123a67390 0x55a123e3f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb576ff5640 0 client.15813 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 --2- 172.21.15.28:0/3856812437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a123a5f1b0 0x55a123e3ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 >> 172.21.15.28:0/3856812437 conn(0x55a123a7c460 msgr2=0x55a123e32390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 shutdown_connections 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 1 -- 172.21.15.28:0/3856812437 wait complete. 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.530+0000 7fb59f9b0740 20 client.15813 trim_cache size 0 max 16384 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 msgr2=0x7fb57c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 0x7fb57c0b8c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb584000c00 tx=0x7fb584005ee0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 msgr2=0x7fb59419fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009fd0 tx=0x7fb58c00b260 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 shutdown_connections 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb57c0b6700 0x7fb57c0b8c10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb594106c10 0x7fb59419fdb0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5941062c0 0x7fb59419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 --2- 172.21.15.28:0/509336429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb594102070 0x7fb59419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 >> 172.21.15.28:0/509336429 conn(0x7fb594079e10 msgr2=0x7fb5941075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 shutdown_connections 2024-09-21T05:47:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:30.531+0000 7fb59943c640 1 -- 172.21.15.28:0/509336429 wait complete. 2024-09-21T05:47:31.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:47:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:31.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:31 smithi028 ceph-mon[31227]: pgmap v340: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:32.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 -- 172.21.15.28:0/201120401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 msgr2=0x7fac50102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 --2- 172.21.15.28:0/201120401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50102e70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fac40009900 tx=0x7fac4002a410 comp rx=0 tx=0).stop 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 -- 172.21.15.28:0/201120401 shutdown_connections 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 --2- 172.21.15.28:0/201120401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac50108cf0 0x7fac5010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 --2- 172.21.15.28:0/201120401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac501083a0 0x7fac501087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 --2- 172.21.15.28:0/201120401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50102e70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 -- 172.21.15.28:0/201120401 >> 172.21.15.28:0/201120401 conn(0x7fac50079730 msgr2=0x7fac50079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 -- 172.21.15.28:0/201120401 shutdown_connections 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.051+0000 7fac55c14640 1 -- 172.21.15.28:0/201120401 wait complete. 2024-09-21T05:47:32.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.053+0000 7fac55c14640 1 Processor -- start 2024-09-21T05:47:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.053+0000 7fac55c14640 1 -- start start 2024-09-21T05:47:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.053+0000 7fac55c14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac55c14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 0x7fac501998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac55c14640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac50108cf0 0x7fac5019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34798/0 (socket says 172.21.15.28:34798) 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4f7fe640 1 -- 172.21.15.28:0/1759057059 learned_addr learned my addr 172.21.15.28:0/1759057059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac50112960 con 0x7fac50102a60 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac501127e0 con 0x7fac501083a0 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac50112ae0 con 0x7fac50108cf0 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4effd640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 0x7fac501998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4ffff640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac50108cf0 0x7fac5019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.054+0000 7fac4effd640 1 -- 172.21.15.28:0/1759057059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac50108cf0 msgr2=0x7fac5019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4effd640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac50108cf0 0x7fac5019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4effd640 1 -- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 msgr2=0x7fac50199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4effd640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4effd640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5019f2c0 con 0x7fac501083a0 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4effd640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 0x7fac501998f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4400d6a0 tx=0x7fac4400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4cff9640 1 -- 172.21.15.28:0/1759057059 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44017070 con 0x7fac501083a0 2024-09-21T05:47:32.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4cff9640 1 -- 172.21.15.28:0/1759057059 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac44009820 con 0x7fac501083a0 2024-09-21T05:47:32.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5019f5b0 con 0x7fac501083a0 2024-09-21T05:47:32.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.055+0000 7fac4cff9640 1 -- 172.21.15.28:0/1759057059 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44004c40 con 0x7fac501083a0 2024-09-21T05:47:32.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.056+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac501a5d10 con 0x7fac501083a0 2024-09-21T05:47:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.059+0000 7fac5c188740 1 Processor -- start 2024-09-21T05:47:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.059+0000 7fac5c188740 1 -- start start 2024-09-21T05:47:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x563908080800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x563908080d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563907ff6ea0 con 0x563907cd9070 2024-09-21T05:47:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563907fe62d0 con 0x563907cb01b0 2024-09-21T05:47:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac5c188740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563907fea0e0 con 0x563907cb8390 2024-09-21T05:47:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4cff9640 1 -- 172.21.15.28:0/1759057059 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fac44005220 con 0x7fac501083a0 2024-09-21T05:47:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34812/0 (socket says 172.21.15.28:34812) 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 -- 172.21.15.28:0/155558364 learned_addr learned my addr 172.21.15.28:0/155558364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 -- 172.21.15.28:0/155558364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 msgr2=0x563908080d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x563908080d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 -- 172.21.15.28:0/155558364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 msgr2=0x563908080800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x563908080800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.060+0000 7fac4ffff640 1 -- 172.21.15.28:0/155558364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563907d44650 con 0x563907cd9070 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.061+0000 7fac4ffff640 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fac48009fd0 tx=0x7fac480118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540cf1cd6b7b6ab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.062+0000 7fac35ffb640 1 -- 172.21.15.28:0/155558364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48022030 con 0x563907cd9070 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.061+0000 7fac4cff9640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 0x7fac380b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.062+0000 7fac35ffb640 1 -- 172.21.15.28:0/155558364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac48003970 con 0x563907cd9070 2024-09-21T05:47:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.062+0000 7fac4cff9640 1 -- 172.21.15.28:0/1759057059 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fac440e6f80 con 0x7fac501083a0 2024-09-21T05:47:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.062+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 0x7fac380b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac35ffb640 1 -- 172.21.15.28:0/155558364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48004520 con 0x563907cd9070 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 0x7fac380b8b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fac40031440 tx=0x7fac40005d50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 -- 172.21.15.28:0/155558364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 msgr2=0x5639080802c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fac48009fd0 tx=0x7fac480118d0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 -- 172.21.15.28:0/155558364 shutdown_connections 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x563908080d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x563908080800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 --2- 172.21.15.28:0/155558364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x5639080802c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 -- 172.21.15.28:0/155558364 >> 172.21.15.28:0/155558364 conn(0x563907ccd460 msgr2=0x56390807e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 -- 172.21.15.28:0/155558364 shutdown_connections 2024-09-21T05:47:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.063+0000 7fac5c188740 1 -- 172.21.15.28:0/155558364 wait complete. 2024-09-21T05:47:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.064+0000 7fac5c188740 1 -- start start 2024-09-21T05:47:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.064+0000 7fac5c188740 1 Processor -- start 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.064+0000 7fac5c188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac5c188740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34818/0 (socket says 172.21.15.28:34818) 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac5c188740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908097e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 -- 172.21.15.28:0/513552718 learned_addr learned my addr 172.21.15.28:0/513552718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563907e47700 con 0x563907cb01b0 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563907f81160 con 0x563907cb8390 2024-09-21T05:47:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56390808eb40 con 0x563907cd9070 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 -- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 msgr2=0x563908097e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908097e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 -- 172.21.15.28:0/513552718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 msgr2=0x56390808be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.065+0000 7fac4f7fe640 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563907faec90 con 0x563907cb01b0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fac40005f70 tx=0x7fac4002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fd5d97da24373b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac5c188740 10 client.0 _ll_register_callbacks cb 0x563907f9a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac48003970 con 0x563907cb01b0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac2effd640 10 client.15825 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac2effd640 1 -- 172.21.15.28:0/513552718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4004c070 con 0x563907cb01b0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.066+0000 7fac2effd640 1 -- 172.21.15.28:0/513552718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fac40005020 con 0x563907cb01b0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac2effd640 1 -- 172.21.15.28:0/513552718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac40035040 con 0x563907cb01b0 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 10 client.15825 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=27}) -- 0x56390809a700 con 0x563907cb01b0 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac2effd640 1 -- 172.21.15.28:0/513552718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fac40057020 con 0x563907cb01b0 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 10 client.15825 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 10 client.15825 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 20 client.15825 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c979f0 0x56390809df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 4 client.15825 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.067+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fac48003970 con 0x563907c979f0 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.068+0000 7fac4effd640 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c979f0 0x56390809df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.068+0000 7fac2effd640 10 client.15825 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.068+0000 7fac4effd640 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c979f0 0x56390809df90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=189e17f403f5236f server_cookie=9b4c0485ca4c4b93 in_seq=0 out_seq=0 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac2effd640 1 -- 172.21.15.28:0/513552718 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fac48003970 con 0x563907c979f0 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac2effd640 10 client.15825 handle_command_reply: tid=0 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 client.15825 shutdown 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 msgr2=0x56390808b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fac40005f70 tx=0x7fac4002a410 comp rx=0 tx=0).stop 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 shutdown_connections 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c979f0 0x56390809df90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908097e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 --2- 172.21.15.28:0/513552718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x56390808b930 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.069+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 >> 172.21.15.28:0/513552718 conn(0x563907ccd460 msgr2=0x5639080834a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.070+0000 7fac2effd640 0 client.15825 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.070+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 shutdown_connections 2024-09-21T05:47:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.070+0000 7fac5c188740 1 -- 172.21.15.28:0/513552718 wait complete. 2024-09-21T05:47:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.070+0000 7fac5c188740 20 client.15825 trim_cache size 0 max 16384 2024-09-21T05:47:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 Processor -- start 2024-09-21T05:47:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 -- start start 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x56390808d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x56390807d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac501ad590 con 0x563907cd9070 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac501af0d0 con 0x563907cb01b0 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac5c188740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac501adc60 con 0x563907cb8390 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34830/0 (socket says 172.21.15.28:34830) 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.097+0000 7fac4ffff640 1 -- 172.21.15.28:0/2942161118 learned_addr learned my addr 172.21.15.28:0/2942161118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 -- 172.21.15.28:0/2942161118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 msgr2=0x56390807d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4effd640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x56390807d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x56390807d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 -- 172.21.15.28:0/2942161118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 msgr2=0x56390808d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x56390808d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x56390808d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 -- 172.21.15.28:0/2942161118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56390808cbf0 con 0x563907cd9070 2024-09-21T05:47:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x56390808d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.098+0000 7fac4ffff640 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fac48006010 tx=0x7fac48016370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d604cf8a539b498e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.099+0000 7fac357fa640 1 -- 172.21.15.28:0/2942161118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48003bf0 con 0x563907cd9070 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.099+0000 7fac357fa640 1 -- 172.21.15.28:0/2942161118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac48003d90 con 0x563907cd9070 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.099+0000 7fac357fa640 1 -- 172.21.15.28:0/2942161118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480108b0 con 0x563907cd9070 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- 172.21.15.28:0/2942161118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 msgr2=0x56390807dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fac48006010 tx=0x7fac48016370 comp rx=0 tx=0).stop 2024-09-21T05:47:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- 172.21.15.28:0/2942161118 shutdown_connections 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cd9070 0x56390807dca0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cb8390 0x56390807d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 --2- 172.21.15.28:0/2942161118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb01b0 0x56390808d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- 172.21.15.28:0/2942161118 >> 172.21.15.28:0/2942161118 conn(0x563907ccd460 msgr2=0x56390808ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- 172.21.15.28:0/2942161118 shutdown_connections 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- 172.21.15.28:0/2942161118 wait complete. 2024-09-21T05:47:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.100+0000 7fac5c188740 1 -- start start 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 Processor -- start 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908090200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac501aea00 con 0x563907cb01b0 2024-09-21T05:47:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac501ae330 con 0x563907cb8390 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac5c188740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac501ab570 con 0x563907cd9070 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34840/0 (socket says 172.21.15.28:34840) 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.101+0000 7fac4f7fe640 1 -- 172.21.15.28:0/2586495084 learned_addr learned my addr 172.21.15.28:0/2586495084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac5c188740 10 client.0 _ll_register_callbacks cb 0x7fac40040050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4effd640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4ffff640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908090200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 -- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 msgr2=0x563908090200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908090200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 -- 172.21.15.28:0/2586495084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 msgr2=0x56390808fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 -- 172.21.15.28:0/2586495084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563908090960 con 0x563907cb01b0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac4f7fe640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fac40031950 tx=0x7fac4003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2343ca5d6226bd5a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.102+0000 7fac2effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40004e30 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fac4003b3d0 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5639080874e0 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac40005610 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac5c188740 10 client.15831 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.103+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5639081088d0 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5639080874e0 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fac40057070 con 0x563907cb01b0 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 10 client.15831 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 10 client.15831 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 20 client.15831 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c6cda0 0x56390810b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 4 client.15831 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563907ff8df0 con 0x563907c6cda0 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.104+0000 7fac4effd640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c6cda0 0x56390810b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac4effd640 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c6cda0 0x56390810b0b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f07d2bb191a5530 server_cookie=ba1e9653f3c73c62 in_seq=0 out_seq=0 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac2effd640 10 client.15831 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac2effd640 1 -- 172.21.15.28:0/2586495084 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563907ff8df0 con 0x563907c6cda0 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac2effd640 10 client.15831 handle_command_reply: tid=0 2024-09-21T05:47:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac5c188740 1 client.15831 shutdown 2024-09-21T05:47:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 msgr2=0x563908094660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.105+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fac40031950 tx=0x7fac4003f000 comp rx=0 tx=0).stop 2024-09-21T05:47:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 shutdown_connections 2024-09-21T05:47:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x563907c6cda0 0x56390810b0b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac2effd640 0 client.15831 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:32.110 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563907cd9070 0x563908090200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563907cb8390 0x56390808fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 --2- 172.21.15.28:0/2586495084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563907cb01b0 0x563908094660 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 >> 172.21.15.28:0/2586495084 conn(0x563907ccd460 msgr2=0x56390808cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 shutdown_connections 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 1 -- 172.21.15.28:0/2586495084 wait complete. 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac5c188740 20 client.15831 trim_cache size 0 max 16384 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.106+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 msgr2=0x7fac380b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 0x7fac380b8b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fac40031440 tx=0x7fac40005d50 comp rx=0 tx=0).stop 2024-09-21T05:47:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 msgr2=0x7fac501998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 0x7fac501998f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4400d6a0 tx=0x7fac4400db60 comp rx=0 tx=0).stop 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 shutdown_connections 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac50108cf0 0x7fac5019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac501083a0 0x7fac501998f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fac380b6680 0x7fac380b8b90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 --2- 172.21.15.28:0/1759057059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac50102a60 0x7fac50199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 >> 172.21.15.28:0/1759057059 conn(0x7fac50079730 msgr2=0x7fac501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 shutdown_connections 2024-09-21T05:47:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:32.107+0000 7fac55c14640 1 -- 172.21.15.28:0/1759057059 wait complete. 2024-09-21T05:47:32.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:31 smithi181 ceph-mon[34819]: pgmap v340: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:32.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:31 smithi129 ceph-mon[34885]: pgmap v340: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 -- 172.21.15.28:0/1284267492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 msgr2=0x7fb0f4102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 --2- 172.21.15.28:0/1284267492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f4102df0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009900 tx=0x7fb0e402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 -- 172.21.15.28:0/1284267492 shutdown_connections 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 --2- 172.21.15.28:0/1284267492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f410f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 --2- 172.21.15.28:0/1284267492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f4108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 --2- 172.21.15.28:0/1284267492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f4102df0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.635+0000 7fb0f9971640 1 -- 172.21.15.28:0/1284267492 >> 172.21.15.28:0/1284267492 conn(0x7fb0f40fe3e0 msgr2=0x7fb0f4100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.636+0000 7fb0f9971640 1 -- 172.21.15.28:0/1284267492 shutdown_connections 2024-09-21T05:47:32.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.636+0000 7fb0f9971640 1 -- 172.21.15.28:0/1284267492 wait complete. 2024-09-21T05:47:32.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.636+0000 7fb0f9971640 1 Processor -- start 2024-09-21T05:47:32.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.636+0000 7fb0f9971640 1 -- start start 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f4113a40 con 0x7fb0f41029e0 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41138c0 con 0x7fb0f4108c60 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f9971640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f4113bc0 con 0x7fb0f4108310 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34854/0 (socket says 172.21.15.28:34854) 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3839765193 learned_addr learned my addr 172.21.15.28:0/3839765193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3839765193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 msgr2=0x7fb0f419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3839765193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 msgr2=0x7fb0f419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3839765193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f41a03b0 con 0x7fb0f41029e0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.637+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005940 tx=0x7fb0e40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f896f640 1 -- 172.21.15.28:0/3839765193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e403f030 con 0x7fb0f41029e0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f41a0640 con 0x7fb0f41029e0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f896f640 1 -- 172.21.15.28:0/3839765193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e40044c0 con 0x7fb0f41029e0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f41a6e30 con 0x7fb0f41029e0 2024-09-21T05:47:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.638+0000 7fb0f896f640 1 -- 172.21.15.28:0/3839765193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4035070 con 0x7fb0f41029e0 2024-09-21T05:47:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.641+0000 7fb0f896f640 1 -- 172.21.15.28:0/3839765193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb0e4042030 con 0x7fb0f41029e0 2024-09-21T05:47:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.641+0000 7fb100522740 1 Processor -- start 2024-09-21T05:47:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb100522740 1 -- start start 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb100522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 0x560f160069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb100522740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16006ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb100522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c404d0 0x560f16007430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb0f896f640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 0x7fb0dc0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 0x560f160069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 0x560f160069b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34866/0 (socket says 172.21.15.28:34866) 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16006ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.642+0000 7fb0f37fe640 1 -- 172.21.15.28:0/944911975 learned_addr learned my addr 172.21.15.28:0/944911975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f896f640 1 -- 172.21.15.28:0/3839765193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb0e4107540 con 0x7fb0f41029e0 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f15f7b6f0 con 0x560f15c611b0 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f15f6ab20 con 0x560f15c382f0 2024-09-21T05:47:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560f15f6e930 con 0x560f15c404d0 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c404d0 0x560f16007430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 0x7fb0dc0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/944911975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c404d0 msgr2=0x560f16007430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c404d0 0x560f16007430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/944911975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 msgr2=0x560f160069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 0x560f160069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/944911975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f15f30530 con 0x560f15c382f0 2024-09-21T05:47:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16006ef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4104c80 tx=0x7fb0e403b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f23231bfb20e247 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.643+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 0x7fb0dc0b8c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec005c40 tx=0x7fb0ec004ab0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.644+0000 7fb0d9ffb640 1 -- 172.21.15.28:0/944911975 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4049070 con 0x560f15c382f0 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.644+0000 7fb0d9ffb640 1 -- 172.21.15.28:0/944911975 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e40ccec0 con 0x560f15c382f0 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.644+0000 7fb0d9ffb640 1 -- 172.21.15.28:0/944911975 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40d6ba0 con 0x560f15c382f0 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 msgr2=0x560f16006ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16006ef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4104c80 tx=0x7fb0e403b000 comp rx=0 tx=0).stop 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 shutdown_connections 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c404d0 0x560f16007430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16006ef0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 --2- 172.21.15.28:0/944911975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c611b0 0x560f160069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 >> 172.21.15.28:0/944911975 conn(0x560f15c55300 msgr2=0x560f160048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 shutdown_connections 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- 172.21.15.28:0/944911975 wait complete. 2024-09-21T05:47:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.645+0000 7fb100522740 1 -- start start 2024-09-21T05:47:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 Processor -- start 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16012060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34876/0 (socket says 172.21.15.28:34876) 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb0f27fc640 1 -- 172.21.15.28:0/1631587136 learned_addr learned my addr 172.21.15.28:0/1631587136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 0x560f1601e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f15ea2ab0 con 0x560f15c404d0 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f15f10730 con 0x560f15c382f0 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560f16014ef0 con 0x560f15c611b0 2024-09-21T05:47:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.646+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16012060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 0x560f1601e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 msgr2=0x560f1601e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 0x560f1601e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb100522740 10 client.0 _ll_register_callbacks cb 0x560f15f646b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/1631587136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 msgr2=0x560f160125a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f15ccc790 con 0x560f15c382f0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 0x560f1601e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0e40ccec0 con 0x560f15c382f0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16012060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e40c9850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ee3c35d9ad10b84 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0d2ffd640 10 client.25457 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/1631587136 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4049040 con 0x560f15c382f0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/1631587136 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb0e41070e0 con 0x560f15c382f0 2024-09-21T05:47:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.647+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/1631587136 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0e4107320 con 0x560f15c382f0 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 10 client.25457 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=27}) -- 0x560f15d45620 con 0x560f15c382f0 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/1631587136 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb0e404c060 con 0x560f15c382f0 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 10 client.25457 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 20 client.25457 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c1fd80 0x560f16023b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 4 client.25457 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f15f7d640 con 0x560f15c1fd80 2024-09-21T05:47:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.648+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c1fd80 0x560f16023b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.649+0000 7fb0d2ffd640 10 client.25457 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.649+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c1fd80 0x560f16023b80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9bb4d6688ed722 server_cookie=36ec9ae7f3aa3ab3 in_seq=0 out_seq=0 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.649+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/1631587136 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560f15f7d640 con 0x560f15c1fd80 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.649+0000 7fb0d2ffd640 10 client.25457 handle_command_reply: tid=0 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 client.25457 shutdown 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 msgr2=0x560f16012060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16012060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e40c9850 comp rx=0 tx=0).stop 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 shutdown_connections 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c1fd80 0x560f16023b80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb0d2ffd640 0 client.25457 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c611b0 0x560f1601e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c404d0 0x560f160125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 --2- 172.21.15.28:0/1631587136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c382f0 0x560f16012060 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 >> 172.21.15.28:0/1631587136 conn(0x560f15c55300 msgr2=0x560f160079e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 shutdown_connections 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 1 -- 172.21.15.28:0/1631587136 wait complete. 2024-09-21T05:47:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.650+0000 7fb100522740 20 client.25457 trim_cache size 0 max 16384 2024-09-21T05:47:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.666+0000 7fb100522740 1 Processor -- start 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.666+0000 7fb100522740 1 -- start start 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.666+0000 7fb100522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c1fd80 0x560f16008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.666+0000 7fb100522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb100522740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f160292f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb100522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000e6d0 con 0x560f15c382f0 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb100522740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0010210 con 0x560f15c404d0 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb100522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000eda0 con 0x560f15c1fd80 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34880/0 (socket says 172.21.15.28:34880) 2024-09-21T05:47:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 -- 172.21.15.28:0/2164828849 learned_addr learned my addr 172.21.15.28:0/2164828849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f160292f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c1fd80 0x560f16008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 -- 172.21.15.28:0/2164828849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c1fd80 msgr2=0x560f16008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c1fd80 0x560f16008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 -- 172.21.15.28:0/2164828849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 msgr2=0x560f160292f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f160292f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 -- 172.21.15.28:0/2164828849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f16010540 con 0x560f15c382f0 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f160292f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8013770 tx=0x7fb0e8004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5da77355c299db1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.667+0000 7fb0d97fa640 1 -- 172.21.15.28:0/2164828849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8013030 con 0x560f15c382f0 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.668+0000 7fb0d97fa640 1 -- 172.21.15.28:0/2164828849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e80131d0 con 0x560f15c382f0 2024-09-21T05:47:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.668+0000 7fb0d97fa640 1 -- 172.21.15.28:0/2164828849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8015370 con 0x560f15c382f0 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- 172.21.15.28:0/2164828849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 msgr2=0x560f160035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8013770 tx=0x7fb0e8004410 comp rx=0 tx=0).stop 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- 172.21.15.28:0/2164828849 shutdown_connections 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f160292f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c382f0 0x560f160035b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 --2- 172.21.15.28:0/2164828849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c1fd80 0x560f16008350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- 172.21.15.28:0/2164828849 >> 172.21.15.28:0/2164828849 conn(0x560f15c55300 msgr2=0x560f16002ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- 172.21.15.28:0/2164828849 shutdown_connections 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- 172.21.15.28:0/2164828849 wait complete. 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.669+0000 7fb100522740 1 -- start start 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 Processor -- start 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c382f0 0x560f160181c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f16085540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000fb40 con 0x560f15c1fd80 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000f470 con 0x560f15c404d0 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000c6b0 con 0x560f15c382f0 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb100522740 10 client.0 _ll_register_callbacks cb 0x7fb0ec009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34888/0 (socket says 172.21.15.28:34888) 2024-09-21T05:47:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3557329607 learned_addr learned my addr 172.21.15.28:0/3557329607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c382f0 0x560f160181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c382f0 msgr2=0x560f160181c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c382f0 0x560f160181c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f37fe640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f16085540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3557329607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 msgr2=0x560f16085540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f16085540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.670+0000 7fb0f2ffd640 1 -- 172.21.15.28:0/3557329607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f16085bb0 con 0x560f15c1fd80 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0f2ffd640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40cd0b0 tx=0x7fb0e4035000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d825f36de695cf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e407d020 con 0x560f15c1fd80 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb0e40cc810 con 0x560f15c1fd80 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0e80131d0 con 0x560f15c1fd80 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0e4049040 con 0x560f15c1fd80 2024-09-21T05:47:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb100522740 10 client.15849 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f16085bb0 con 0x560f15c1fd80 2024-09-21T05:47:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.671+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e80131d0 con 0x560f15c1fd80 2024-09-21T05:47:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb0e40c7ab0 con 0x560f15c1fd80 2024-09-21T05:47:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 10 client.15849 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 10 client.15849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 20 client.15849 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c611b0 0x560f16090520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 4 client.15849 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f15f790f0 con 0x560f15c611b0 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.672+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c611b0 0x560f16090520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.673+0000 7fb0f27fc640 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c611b0 0x560f16090520 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ccff11ebc7f30e server_cookie=8cb3201bd537f7f6 in_seq=0 out_seq=0 2024-09-21T05:47:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.673+0000 7fb0d2ffd640 10 client.15849 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb0d2ffd640 1 -- 172.21.15.28:0/3557329607 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f15f790f0 con 0x560f15c611b0 2024-09-21T05:47:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb0d2ffd640 10 client.15849 handle_command_reply: tid=0 2024-09-21T05:47:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 client.15849 shutdown 2024-09-21T05:47:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 msgr2=0x560f16017c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40cd0b0 tx=0x7fb0e4035000 comp rx=0 tx=0).stop 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 shutdown_connections 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f15c611b0 0x560f16090520 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f15c404d0 0x560f16085540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb0d2ffd640 0 client.15849 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f15c382f0 0x560f160181c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 --2- 172.21.15.28:0/3557329607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f15c1fd80 0x560f16017c20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 >> 172.21.15.28:0/3557329607 conn(0x560f15c55300 msgr2=0x560f16008120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 shutdown_connections 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 1 -- 172.21.15.28:0/3557329607 wait complete. 2024-09-21T05:47:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.679+0000 7fb100522740 20 client.15849 trim_cache size 0 max 16384 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 msgr2=0x7fb0dc0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 0x7fb0dc0b8c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec005c40 tx=0x7fb0ec004ab0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 msgr2=0x7fb0f419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005940 tx=0x7fb0e40040c0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 shutdown_connections 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0f4108c60 0x7fb0f419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0f4108310 0x7fb0f419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb0dc0b6700 0x7fb0dc0b8c10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 --2- 172.21.15.28:0/3839765193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f41029e0 0x7fb0f419a480 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 >> 172.21.15.28:0/3839765193 conn(0x7fb0f40fe3e0 msgr2=0x7fb0f4107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 shutdown_connections 2024-09-21T05:47:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:32.680+0000 7fb0f9971640 1 -- 172.21.15.28:0/3839765193 wait complete. 2024-09-21T05:47:32.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:47:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 -- 172.21.15.28:0/86381726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 msgr2=0x7f6d84112610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/86381726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d84112610 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009ca0 tx=0x7f6d7c02a950 comp rx=0 tx=0).stop 2024-09-21T05:47:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 -- 172.21.15.28:0/86381726 shutdown_connections 2024-09-21T05:47:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/86381726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d84112610 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/86381726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102f90 0x7f6d8410fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/86381726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102640 0x7f6d84102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 -- 172.21.15.28:0/86381726 >> 172.21.15.28:0/86381726 conn(0x7f6d84079730 msgr2=0x7f6d84079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 -- 172.21.15.28:0/86381726 shutdown_connections 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.502+0000 7f6d8b22b640 1 -- 172.21.15.28:0/86381726 wait complete. 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.503+0000 7f6d8b22b640 1 Processor -- start 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.503+0000 7f6d8b22b640 1 -- start start 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.503+0000 7f6d8b22b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102640 0x7f6d8419c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.503+0000 7f6d8b22b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 0x7f6d8419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.503+0000 7f6d8b22b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d841a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d841a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d841a1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34916/0 (socket says 172.21.15.28:34916) 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d897c8640 1 -- 172.21.15.28:0/2850824426 learned_addr learned my addr 172.21.15.28:0/2850824426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d84115c60 con 0x7f6d841101f0 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d84115ae0 con 0x7f6d84102f90 2024-09-21T05:47:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6d84115de0 con 0x7f6d84102640 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 0x7f6d8419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102640 0x7f6d8419c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 -- 172.21.15.28:0/2850824426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102640 msgr2=0x7f6d8419c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102640 0x7f6d8419c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 -- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 msgr2=0x7f6d841a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d841a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.504+0000 7f6d83fff640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d841a2540 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d83fff640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 0x7f6d8419cb70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9f0 tx=0x7f6d7000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d81ffb640 1 -- 172.21.15.28:0/2850824426 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000ccc0 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d81ffb640 1 -- 172.21.15.28:0/2850824426 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d700108e0 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d841a2830 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d81ffb640 1 -- 172.21.15.28:0/2850824426 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70018880 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.505+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d841a8f90 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.508+0000 7f6d81ffb640 1 -- 172.21.15.28:0/2850824426 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f6d70010430 con 0x7f6d84102f90 2024-09-21T05:47:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 Processor -- start 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 -- start start 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 0x55eb235c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb2353bea0 con 0x55eb231fd390 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eb2352b2d0 con 0x55eb231f51b0 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d9179f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eb2352f0e0 con 0x55eb2321e070 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.509+0000 7f6d81ffb640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 0x7f6d740b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d81ffb640 1 -- 172.21.15.28:0/2850824426 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6d700e7a90 con 0x7f6d84102f90 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 0x55eb235c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d897c8640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 0x7f6d740b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34918/0 (socket says 172.21.15.28:34918) 2024-09-21T05:47:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 -- 172.21.15.28:0/1250244339 learned_addr learned my addr 172.21.15.28:0/1250244339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 -- 172.21.15.28:0/1250244339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 msgr2=0x55eb235c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 0x55eb235c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 -- 172.21.15.28:0/1250244339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 msgr2=0x55eb235c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.510+0000 7f6d83fff640 1 -- 172.21.15.28:0/1250244339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb23289650 con 0x55eb231fd390 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d897c8640 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 0x55eb235c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d83fff640 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6d700ad8e0 tx=0x7f6d700abb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea4b57d447f340e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d897c8640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 0x7f6d740b8b90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009a60 tx=0x7f6d7c009380 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d6effd640 1 -- 172.21.15.28:0/1250244339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d700b2380 con 0x55eb231fd390 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d6effd640 1 -- 172.21.15.28:0/1250244339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d700b72c0 con 0x55eb231fd390 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.511+0000 7f6d6effd640 1 -- 172.21.15.28:0/1250244339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d700a9bc0 con 0x55eb231fd390 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- 172.21.15.28:0/1250244339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 msgr2=0x55eb235c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6d700ad8e0 tx=0x7f6d700abb00 comp rx=0 tx=0).stop 2024-09-21T05:47:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- 172.21.15.28:0/1250244339 shutdown_connections 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c6120 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1250244339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb2321e070 0x55eb235c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- 172.21.15.28:0/1250244339 >> 172.21.15.28:0/1250244339 conn(0x55eb23212460 msgr2=0x55eb235c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- 172.21.15.28:0/1250244339 shutdown_connections 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- 172.21.15.28:0/1250244339 wait complete. 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.513+0000 7f6d9179f740 1 -- start start 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 Processor -- start 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231fd390 0x55eb235d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb2338c700 con 0x55eb2321e070 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eb234c6160 con 0x55eb231f51b0 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.514+0000 7f6d9179f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eb235d3e20 con 0x55eb231fd390 2024-09-21T05:47:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34930/0 (socket says 172.21.15.28:34930) 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 -- 172.21.15.28:0/3896791052 learned_addr learned my addr 172.21.15.28:0/3896791052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 -- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231fd390 msgr2=0x55eb235d1230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d83fff640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231fd390 0x55eb235d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231fd390 0x55eb235d1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 -- 172.21.15.28:0/3896791052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 msgr2=0x55eb235d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 -- 172.21.15.28:0/3896791052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb234f3c90 con 0x55eb2321e070 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d9179f740 10 client.0 _ll_register_callbacks cb 0x55eb234df9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d700b72c0 con 0x55eb2321e070 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d897c8640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02ff20 tx=0x7f6d7c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6510ac02c236d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.515+0000 7f6d63fff640 10 client.15861 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d63fff640 1 -- 172.21.15.28:0/3896791052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c044040 con 0x55eb2321e070 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d63fff640 1 -- 172.21.15.28:0/3896791052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6d7c004040 con 0x55eb2321e070 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d63fff640 1 -- 172.21.15.28:0/3896791052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d7c034040 con 0x55eb2321e070 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 10 client.15861 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55eb235df9e0 con 0x55eb2321e070 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d63fff640 1 -- 172.21.15.28:0/3896791052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6d7c05e020 con 0x55eb2321e070 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 10 client.15861 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 10 client.15861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 20 client.15861 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb231dc9f0 0x55eb235e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.516+0000 7f6d9179f740 4 client.15861 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.517+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb2353ddf0 con 0x55eb231dc9f0 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.517+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb231dc9f0 0x55eb235e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.517+0000 7f6d63fff640 10 client.15861 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:33.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.517+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb231dc9f0 0x55eb235e2e60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e77c41d9ffa15699 server_cookie=98cb97ccd98a3088 in_seq=0 out_seq=0 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.518+0000 7f6d63fff640 1 -- 172.21.15.28:0/3896791052 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55eb2353ddf0 con 0x55eb231dc9f0 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.518+0000 7f6d63fff640 10 client.15861 handle_command_reply: tid=0 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.518+0000 7f6d9179f740 1 client.15861 shutdown 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.518+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 msgr2=0x55eb235dd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.518+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02ff20 tx=0x7f6d7c0095c0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 shutdown_connections 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb231dc9f0 0x55eb235e2e60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d63fff640 0 client.15861 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb2321e070 0x55eb235dd110 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231fd390 0x55eb235d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3896791052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 >> 172.21.15.28:0/3896791052 conn(0x55eb23212460 msgr2=0x55eb235c8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 shutdown_connections 2024-09-21T05:47:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 1 -- 172.21.15.28:0/3896791052 wait complete. 2024-09-21T05:47:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.519+0000 7f6d9179f740 20 client.15861 trim_cache size 0 max 16384 2024-09-21T05:47:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.545+0000 7f6d9179f740 1 Processor -- start 2024-09-21T05:47:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 -- start start 2024-09-21T05:47:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841b1870 con 0x55eb231fd390 2024-09-21T05:47:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841b33b0 con 0x55eb231f51b0 2024-09-21T05:47:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d88fc7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d9179f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841b1f40 con 0x55eb231dc9f0 2024-09-21T05:47:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.546+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:48776/0 (socket says 172.21.15.28:48776) 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d897c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34944/0 (socket says 172.21.15.28:34944) 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 -- 172.21.15.28:0/1754773308 learned_addr learned my addr 172.21.15.28:0/1754773308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 -- 172.21.15.28:0/1754773308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 msgr2=0x55eb235c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 -- 172.21.15.28:0/1754773308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 msgr2=0x55eb235c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 -- 172.21.15.28:0/1754773308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb235c7990 con 0x55eb231dc9f0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.547+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800b9e0 tx=0x7f6d7800b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aad69e1135073b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.548+0000 7f6d6e7fc640 1 -- 172.21.15.28:0/1754773308 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78003ed0 con 0x55eb231dc9f0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.548+0000 7f6d6e7fc640 1 -- 172.21.15.28:0/1754773308 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d7801f030 con 0x55eb231dc9f0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.548+0000 7f6d6e7fc640 1 -- 172.21.15.28:0/1754773308 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78011730 con 0x55eb231dc9f0 2024-09-21T05:47:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- 172.21.15.28:0/1754773308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 msgr2=0x55eb235d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800b9e0 tx=0x7f6d7800b390 comp rx=0 tx=0).stop 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- 172.21.15.28:0/1754773308 shutdown_connections 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb235c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 --2- 172.21.15.28:0/1754773308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d2ce0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- 172.21.15.28:0/1754773308 >> 172.21.15.28:0/1754773308 conn(0x55eb23212460 msgr2=0x55eb235c7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- 172.21.15.28:0/1754773308 shutdown_connections 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- 172.21.15.28:0/1754773308 wait complete. 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.549+0000 7f6d9179f740 1 -- start start 2024-09-21T05:47:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 Processor -- start 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb2364c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841b2ce0 con 0x55eb231fd390 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841b2610 con 0x55eb231f51b0 2024-09-21T05:47:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.550+0000 7f6d9179f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6d841af8a0 con 0x55eb231dc9f0 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d9179f740 10 client.0 _ll_register_callbacks cb 0x7f6d7c02f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60100/0 (socket says 172.21.15.28:60100) 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 -- 172.21.15.28:0/3709421382 learned_addr learned my addr 172.21.15.28:0/3709421382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d897c8640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb2364c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 -- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 msgr2=0x55eb235d5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 -- 172.21.15.28:0/3709421382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 msgr2=0x55eb2364c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb2364c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 -- 172.21.15.28:0/3709421382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb2364cde0 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d83fff640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d700d10d0 tx=0x7f6d700aba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5024b2b9f2123092 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.551+0000 7f6d63fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d700a9fa0 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f6d700e5420 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d7801f030 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d700e5620 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 10 client.25475 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb2364e060 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d70098060 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6d700aebd0 con 0x55eb231f51b0 2024-09-21T05:47:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 10 client.25475 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 10 client.25475 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 20 client.25475 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb2321e070 0x55eb236503c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 4 client.25475 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.552+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6d7801f030 con 0x55eb2321e070 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.553+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb2321e070 0x55eb236503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.553+0000 7f6d63fff640 10 client.25475 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.553+0000 7f6d88fc7640 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb2321e070 0x55eb236503c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78571a595bab2e0e server_cookie=2bdc97fe0f8b99b6 in_seq=0 out_seq=0 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d63fff640 1 -- 172.21.15.28:0/3709421382 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6d7801f030 con 0x55eb2321e070 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d63fff640 10 client.25475 handle_command_reply: tid=0 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 client.25475 shutdown 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 msgr2=0x55eb235d5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d700d10d0 tx=0x7f6d700aba90 comp rx=0 tx=0).stop 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 shutdown_connections 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55eb2321e070 0x55eb236503c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb231fd390 0x55eb2364c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb231f51b0 0x55eb235d5d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 --2- 172.21.15.28:0/3709421382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb231dc9f0 0x55eb235d5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d63fff640 0 client.25475 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 >> 172.21.15.28:0/3709421382 conn(0x55eb23212460 msgr2=0x55eb235ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 shutdown_connections 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 1 -- 172.21.15.28:0/3709421382 wait complete. 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.554+0000 7f6d9179f740 20 client.25475 trim_cache size 0 max 16384 2024-09-21T05:47:33.560 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 msgr2=0x7f6d740b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 0x7f6d740b8b90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009a60 tx=0x7f6d7c009380 comp rx=0 tx=0).stop 2024-09-21T05:47:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 msgr2=0x7f6d8419cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 0x7f6d8419cb70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9f0 tx=0x7f6d7000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 shutdown_connections 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f6d740b6680 0x7f6d740b8b90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d841101f0 0x7f6d841a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.555+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d84102f90 0x7f6d8419cb70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.556+0000 7f6d8b22b640 1 --2- 172.21.15.28:0/2850824426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d84102640 0x7f6d8419c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.556+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 >> 172.21.15.28:0/2850824426 conn(0x7f6d84079730 msgr2=0x7f6d84109d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.556+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 shutdown_connections 2024-09-21T05:47:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:33.556+0000 7f6d8b22b640 1 -- 172.21.15.28:0/2850824426 wait complete. 2024-09-21T05:47:34.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:33 smithi181 ceph-mon[34819]: pgmap v341: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-09-21T05:47:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:33 smithi028 ceph-mon[31227]: pgmap v341: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-09-21T05:47:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:33 smithi129 ceph-mon[34885]: pgmap v341: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-09-21T05:47:34.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:47:34.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 -- 172.21.15.28:0/3976833779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c1094b0 msgr2=0x7fc94c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 --2- 172.21.15.28:0/3976833779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c1094b0 0x7fc94c1098c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc940009900 tx=0x7fc94002a410 comp rx=0 tx=0).stop 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 -- 172.21.15.28:0/3976833779 shutdown_connections 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 --2- 172.21.15.28:0/3976833779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c109e00 0x7fc94c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 --2- 172.21.15.28:0/3976833779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c1094b0 0x7fc94c1098c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 --2- 172.21.15.28:0/3976833779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 0x7fc94c103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 -- 172.21.15.28:0/3976833779 >> 172.21.15.28:0/3976833779 conn(0x7fc94c07b910 msgr2=0x7fc94c07bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 -- 172.21.15.28:0/3976833779 shutdown_connections 2024-09-21T05:47:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.172+0000 7fc951766640 1 -- 172.21.15.28:0/3976833779 wait complete. 2024-09-21T05:47:35.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.173+0000 7fc951766640 1 Processor -- start 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.173+0000 7fc951766640 1 -- start start 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 0x7fc94c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c1094b0 0x7fc94c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c113ac0 con 0x7fc94c109e00 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c113940 con 0x7fc94c1094b0 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc951766640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c113c40 con 0x7fc94c103bc0 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34964/0 (socket says 172.21.15.28:34964) 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94b7fe640 1 -- 172.21.15.28:0/1305990701 learned_addr learned my addr 172.21.15.28:0/1305990701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94affd640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 0x7fc94c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c1094b0 0x7fc94c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.174+0000 7fc94b7fe640 1 -- 172.21.15.28:0/1305990701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 msgr2=0x7fc94c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 0x7fc94c19c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc94b7fe640 1 -- 172.21.15.28:0/1305990701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c1094b0 msgr2=0x7fc94c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c1094b0 0x7fc94c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc94b7fe640 1 -- 172.21.15.28:0/1305990701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c1a26c0 con 0x7fc94c109e00 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc94400eaa0 tx=0x7fc94400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc933fff640 1 -- 172.21.15.28:0/1305990701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94400cdb0 con 0x7fc94c109e00 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc933fff640 1 -- 172.21.15.28:0/1305990701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc944005020 con 0x7fc94c109e00 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.175+0000 7fc933fff640 1 -- 172.21.15.28:0/1305990701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9440116d0 con 0x7fc94c109e00 2024-09-21T05:47:35.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.176+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c1a29b0 con 0x7fc94c109e00 2024-09-21T05:47:35.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.177+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c1a9110 con 0x7fc94c109e00 2024-09-21T05:47:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.179+0000 7fc933fff640 1 -- 172.21.15.28:0/1305990701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc9440051d0 con 0x7fc94c109e00 2024-09-21T05:47:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.180+0000 7fc933fff640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 0x7fc9340b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.180+0000 7fc933fff640 1 -- 172.21.15.28:0/1305990701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc9440e7500 con 0x7fc94c109e00 2024-09-21T05:47:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.180+0000 7fc94affd640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 0x7fc9340b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.180+0000 7fc957cda740 1 Processor -- start 2024-09-21T05:47:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.181+0000 7fc957cda740 1 -- start start 2024-09-21T05:47:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.181+0000 7fc94affd640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 0x7fc9340b8c90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005fd0 tx=0x7fc93c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.181+0000 7fc957cda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 0x55f8bd9a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.181+0000 7fc957cda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc957cda740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc957cda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd91aea0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 0x55f8bd9a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc957cda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd90a2d0 con 0x55f8bd5fd070 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 0x55f8bd9a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60120/0 (socket says 172.21.15.28:60120) 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:48808/0 (socket says 172.21.15.28:48808) 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 -- 172.21.15.28:0/2812163942 learned_addr learned my addr 172.21.15.28:0/2812163942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd90e0e0 con 0x55f8bd5dc390 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 -- 172.21.15.28:0/2812163942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 msgr2=0x55f8bd9a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 0x55f8bd9a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.182+0000 7fc94b7fe640 1 -- 172.21.15.28:0/2812163942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 msgr2=0x55f8bd9a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc94b7fe640 1 -- 172.21.15.28:0/2812163942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bd668650 con 0x55f8bd5dc390 2024-09-21T05:47:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9440a98e0 tx=0x7fc9440ad210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fc110451923bdd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc9317fa640 1 -- 172.21.15.28:0/2812163942 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94401f070 con 0x55f8bd5dc390 2024-09-21T05:47:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc9317fa640 1 -- 172.21.15.28:0/2812163942 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9440b1df0 con 0x55f8bd5dc390 2024-09-21T05:47:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.183+0000 7fc9317fa640 1 -- 172.21.15.28:0/2812163942 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9440b6040 con 0x55f8bd5dc390 2024-09-21T05:47:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.184+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 msgr2=0x55f8bd9a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.184+0000 7fc957cda740 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9440a98e0 tx=0x7fc9440ad210 comp rx=0 tx=0).stop 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.184+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 shutdown_connections 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a5120 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 --2- 172.21.15.28:0/2812163942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5fd070 0x55f8bd9a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 >> 172.21.15.28:0/2812163942 conn(0x55f8bd5f1460 msgr2=0x55f8bd9a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 shutdown_connections 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 -- 172.21.15.28:0/2812163942 wait complete. 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 -- start start 2024-09-21T05:47:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.185+0000 7fc957cda740 1 Processor -- start 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5fd070 0x55f8bd9bc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc94a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc94a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34984/0 (socket says 172.21.15.28:34984) 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc94a7fc640 1 -- 172.21.15.28:0/1155232634 learned_addr learned my addr 172.21.15.28:0/1155232634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd76b700 con 0x55f8bd5dc390 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd8a5160 con 0x55f8bd5d41b0 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f8bd9b2e80 con 0x55f8bd5fd070 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc94affd640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.186+0000 7fc957cda740 10 client.0 _ll_register_callbacks cb 0x55f8bd8be9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 -- 172.21.15.28:0/1155232634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5fd070 msgr2=0x55f8bd9bc170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5fd070 0x55f8bd9bc170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 -- 172.21.15.28:0/1155232634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 msgr2=0x55f8bd9b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8bd8d2c90 con 0x55f8bd5d41b0 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc94affd640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9afdb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0058c0 tx=0x7fc93c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef6782372bb42f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc92eff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0108d0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc93c010a70 con 0x55f8bd5d41b0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.187+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9440b1df0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc93c010cb0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc957cda740 10 client.25487 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8bd9beba0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc93c017030 con 0x55f8bd5d41b0 2024-09-21T05:47:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc93c02e020 con 0x55f8bd5d41b0 2024-09-21T05:47:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc957cda740 10 client.25487 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.188+0000 7fc957cda740 10 client.25487 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc957cda740 20 client.25487 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5bb9f0 0x55f8bd9c2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc957cda740 4 client.25487 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc9440b1df0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5bb9f0 0x55f8bd9c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5bb9f0 0x55f8bd9c2140 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54c615c9de6a07f2 server_cookie=8cec1c798ed4c3b7 in_seq=0 out_seq=0 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.189+0000 7fc92eff5640 10 client.25487 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.190+0000 7fc92eff5640 1 -- 172.21.15.28:0/1155232634 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fc9440b1df0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.190+0000 7fc92eff5640 10 client.25487 handle_command_reply: tid=0 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.190+0000 7fc957cda740 1 client.25487 shutdown 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 msgr2=0x55f8bd9afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9afdb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0058c0 tx=0x7fc93c0046c0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 shutdown_connections 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5bb9f0 0x55f8bd9c2140 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5fd070 0x55f8bd9bc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5dc390 0x55f8bd9b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 --2- 172.21.15.28:0/1155232634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9afdb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc92eff5640 0 client.25487 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 >> 172.21.15.28:0/1155232634 conn(0x55f8bd5f1460 msgr2=0x55f8bd9a7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 shutdown_connections 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 1 -- 172.21.15.28:0/1155232634 wait complete. 2024-09-21T05:47:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.191+0000 7fc957cda740 20 client.25487 trim_cache size 0 max 16384 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.204+0000 7fc957cda740 1 Processor -- start 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 -- start start 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1b1960 con 0x55f8bd5d41b0 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1b34a0 con 0x55f8bd5dc390 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc957cda740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1b2030 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34998/0 (socket says 172.21.15.28:34998) 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 -- 172.21.15.28:0/2341089968 learned_addr learned my addr 172.21.15.28:0/2341089968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 -- 172.21.15.28:0/2341089968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5bb9f0 msgr2=0x55f8bd9b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 -- 172.21.15.28:0/2341089968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 msgr2=0x55f8bd9a7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.205+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.206+0000 7fc94a7fc640 1 -- 172.21.15.28:0/2341089968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bd9ab640 con 0x55f8bd5d41b0 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.206+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc940005940 tx=0x7fc940004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea89e4d05c28dba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.206+0000 7fc930ff9640 1 -- 172.21.15.28:0/2341089968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940035040 con 0x55f8bd5d41b0 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.206+0000 7fc930ff9640 1 -- 172.21.15.28:0/2341089968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9400044d0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.206+0000 7fc930ff9640 1 -- 172.21.15.28:0/2341089968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9400049b0 con 0x55f8bd5d41b0 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 -- 172.21.15.28:0/2341089968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 msgr2=0x55f8bd9a77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc940005940 tx=0x7fc940004360 comp rx=0 tx=0).stop 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 -- 172.21.15.28:0/2341089968 shutdown_connections 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bd9a7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9a77a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 --2- 172.21.15.28:0/2341089968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 -- 172.21.15.28:0/2341089968 >> 172.21.15.28:0/2341089968 conn(0x55f8bd5f1460 msgr2=0x55f8bd9afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 -- 172.21.15.28:0/2341089968 shutdown_connections 2024-09-21T05:47:35.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.207+0000 7fc957cda740 1 -- 172.21.15.28:0/2341089968 wait complete. 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.208+0000 7fc957cda740 1 -- start start 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.208+0000 7fc957cda740 1 Processor -- start 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.208+0000 7fc957cda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.208+0000 7fc957cda740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.208+0000 7fc957cda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bda2bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc957cda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1b2dd0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35012/0 (socket says 172.21.15.28:35012) 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 -- 172.21.15.28:0/1984841897 learned_addr learned my addr 172.21.15.28:0/1984841897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1b2700 con 0x55f8bd5dc390 2024-09-21T05:47:35.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94b7fe640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bda2bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c1af940 con 0x55f8bd5d41b0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc957cda740 10 client.0 _ll_register_callbacks cb 0x7fc93c011540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 -- 172.21.15.28:0/1984841897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5d41b0 msgr2=0x55f8bd9b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 -- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 msgr2=0x55f8bda2bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bda2bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.209+0000 7fc94affd640 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8bda2c040 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc94affd640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc93c027610 tx=0x7fc93c011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c716a6a323d4f0d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc92eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c027a40 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc93c027be0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9400044d0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc93c027de0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc957cda740 10 client.15885 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8bda2d0e0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.210+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9400044d0 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc93c02e070 con 0x55f8bd5bb9f0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 10 client.15885 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 10 client.15885 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 20 client.15885 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5fd070 0x55f8bda2f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 4 client.15885 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8bd91cdf0 con 0x55f8bd5fd070 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.211+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5fd070 0x55f8bda2f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc94a7fc640 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5fd070 0x55f8bda2f8a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6e278d1814d8df7 server_cookie=bf1e6f2ad1b1f4a9 in_seq=0 out_seq=0 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc92eff5640 10 client.15885 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc92eff5640 1 -- 172.21.15.28:0/1984841897 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8bd91cdf0 con 0x55f8bd5fd070 2024-09-21T05:47:35.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc92eff5640 10 client.15885 handle_command_reply: tid=0 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc957cda740 1 client.15885 shutdown 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 msgr2=0x55f8bd9b4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc93c027610 tx=0x7fc93c011e10 comp rx=0 tx=0).stop 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 shutdown_connections 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.212+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f8bd5fd070 0x55f8bda2f8a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f8bd5dc390 0x55f8bda2bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc92eff5640 0 client.15885 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f8bd5d41b0 0x55f8bd9b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 --2- 172.21.15.28:0/1984841897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8bd5bb9f0 0x55f8bd9b4470 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 >> 172.21.15.28:0/1984841897 conn(0x55f8bd5f1460 msgr2=0x55f8bd9a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 shutdown_connections 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 1 -- 172.21.15.28:0/1984841897 wait complete. 2024-09-21T05:47:35.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.213+0000 7fc957cda740 20 client.15885 trim_cache size 0 max 16384 2024-09-21T05:47:35.219 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:35.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 msgr2=0x7fc9340b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 0x7fc9340b8c90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005fd0 tx=0x7fc93c005850 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 msgr2=0x7fc94c1a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc94400eaa0 tx=0x7fc94400ef60 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 shutdown_connections 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc9340b6780 0x7fc9340b8c90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94c109e00 0x7fc94c1a1f90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c1094b0 0x7fc94c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 --2- 172.21.15.28:0/1305990701 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc94c103bc0 0x7fc94c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 >> 172.21.15.28:0/1305990701 conn(0x7fc94c07b910 msgr2=0x7fc94c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 shutdown_connections 2024-09-21T05:47:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:35.214+0000 7fc951766640 1 -- 172.21.15.28:0/1305990701 wait complete. 2024-09-21T05:47:35.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:47:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:35 smithi028 ceph-mon[31227]: pgmap v342: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:47:36.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:35 smithi181 ceph-mon[34819]: pgmap v342: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:47:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:35 smithi129 ceph-mon[34885]: pgmap v342: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 -- 172.21.15.28:0/3644415271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 msgr2=0x7fafc8115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 --2- 172.21.15.28:0/3644415271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc8115fd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009900 tx=0x7fafbc02a410 comp rx=0 tx=0).stop 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 -- 172.21.15.28:0/3644415271 shutdown_connections 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 --2- 172.21.15.28:0/3644415271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc8118b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 --2- 172.21.15.28:0/3644415271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc8115fd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 --2- 172.21.15.28:0/3644415271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 0x7fafc810a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.887+0000 7fafce8a8640 1 -- 172.21.15.28:0/3644415271 >> 172.21.15.28:0/3644415271 conn(0x7fafc8070ec0 msgr2=0x7fafc80712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.888+0000 7fafce8a8640 1 -- 172.21.15.28:0/3644415271 shutdown_connections 2024-09-21T05:47:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.888+0000 7fafce8a8640 1 -- 172.21.15.28:0/3644415271 wait complete. 2024-09-21T05:47:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.889+0000 7fafce8a8640 1 Processor -- start 2024-09-21T05:47:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.889+0000 7fafce8a8640 1 -- start start 2024-09-21T05:47:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.890+0000 7fafce8a8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 0x7fafc81a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.890+0000 7fafce8a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.890+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41632/0 (socket says 172.21.15.28:41632) 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafc77fe640 1 -- 172.21.15.28:0/4223604652 learned_addr learned my addr 172.21.15.28:0/4223604652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc81a8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafc811c2d0 con 0x7fafc810aee0 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc811c150 con 0x7fafc8116640 2024-09-21T05:47:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafc811c450 con 0x7fafc810a590 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafcce45640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc81a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.891+0000 7fafc7fff640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 0x7fafc81a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 -- 172.21.15.28:0/4223604652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 msgr2=0x7fafc81a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 0x7fafc81a2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 -- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 msgr2=0x7fafc81a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc81a8a30 con 0x7fafc8116640 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafc77fe640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafcce45640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc81a8300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fafc000e940 tx=0x7fafc000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafc57fa640 1 -- 172.21.15.28:0/4223604652 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0004d20 con 0x7fafc8116640 2024-09-21T05:47:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafc57fa640 1 -- 172.21.15.28:0/4223604652 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc0005020 con 0x7fafc8116640 2024-09-21T05:47:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.892+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc81a8d20 con 0x7fafc8116640 2024-09-21T05:47:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.893+0000 7fafc57fa640 1 -- 172.21.15.28:0/4223604652 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00116d0 con 0x7fafc8116640 2024-09-21T05:47:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.893+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc81af480 con 0x7fafc8116640 2024-09-21T05:47:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.896+0000 7fafd4e1c740 1 Processor -- start 2024-09-21T05:47:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.897+0000 7fafc57fa640 1 -- 172.21.15.28:0/4223604652 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fafc0005460 con 0x7fafc8116640 2024-09-21T05:47:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.897+0000 7fafd4e1c740 1 -- start start 2024-09-21T05:47:36.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafc57fa640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 0x7fafb00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bf2070 0x558d20f97100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bc91b0 0x558d20f97680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d20f0fea0 con 0x558d20bc91b0 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558d20eff2d0 con 0x558d20bd1390 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.898+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558d20f030e0 con 0x558d20bf2070 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55746/0 (socket says 172.21.15.28:55746) 2024-09-21T05:47:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 -- 172.21.15.28:0/3959730686 learned_addr learned my addr 172.21.15.28:0/3959730686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafc7fff640 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bf2070 0x558d20f97100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafc57fa640 1 -- 172.21.15.28:0/4223604652 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fafc00e7730 con 0x7fafc8116640 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 -- 172.21.15.28:0/3959730686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bf2070 msgr2=0x558d20f97100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bf2070 0x558d20f97100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 -- 172.21.15.28:0/3959730686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bc91b0 msgr2=0x558d20f97680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:36.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bc91b0 0x558d20f97680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 -- 172.21.15.28:0/3959730686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d20c5d650 con 0x558d20bd1390 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafc77fe640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 0x7fafb00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafcce45640 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafc00a9bc0 tx=0x7fafc00791a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e9c8e83762edd90 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.899+0000 7fafae7fc640 1 -- 172.21.15.28:0/3959730686 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc001f050 con 0x558d20bd1390 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.900+0000 7fafae7fc640 1 -- 172.21.15.28:0/3959730686 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc00ad100 con 0x558d20bd1390 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.900+0000 7fafae7fc640 1 -- 172.21.15.28:0/3959730686 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00aeaa0 con 0x558d20bd1390 2024-09-21T05:47:36.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3959730686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 msgr2=0x558d20fa9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafc00a9bc0 tx=0x7fafc00791a0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3959730686 shutdown_connections 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d20fa9a50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bc91b0 0x558d20f97680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3959730686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bf2070 0x558d20f97100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3959730686 >> 172.21.15.28:0/3959730686 conn(0x558d20be6460 msgr2=0x558d20f99620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafc77fe640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 0x7fafb00b8c10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005f70 tx=0x7fafbc031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3959730686 shutdown_connections 2024-09-21T05:47:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.901+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3959730686 wait complete. 2024-09-21T05:47:36.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.902+0000 7fafd4e1c740 1 -- start start 2024-09-21T05:47:36.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.902+0000 7fafd4e1c740 1 Processor -- start 2024-09-21T05:47:36.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.903+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.903+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 0x558d20fae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fb3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d20d60700 con 0x558d20bf2070 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558d20e9a160 con 0x558d20bb09f0 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:39846/0 (socket says 172.21.15.28:39846) 2024-09-21T05:47:36.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558d20faa490 con 0x558d20bd1390 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 -- 172.21.15.28:0/3674140073 learned_addr learned my addr 172.21.15.28:0/3674140073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 0x558d20fae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafd4e1c740 10 client.0 _ll_register_callbacks cb 0x558d20eb39e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafcce45640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fb3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 -- 172.21.15.28:0/3674140073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 msgr2=0x558d20fb3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fb3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 -- 172.21.15.28:0/3674140073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 msgr2=0x558d20fae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 0x558d20fae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.904+0000 7fafc7fff640 1 -- 172.21.15.28:0/3674140073 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d20ec7c90 con 0x558d20bd1390 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafc77fe640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 0x558d20fae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafc7fff640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fafb8012770 tx=0x7fafb8012c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd635de717d222f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafa77fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb800f9a0 con 0x558d20bd1390 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fafb800fb40 con 0x558d20bd1390 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.905+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc00ad100 con 0x558d20bd1390 2024-09-21T05:47:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafb800fd40 con 0x558d20bd1390 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 10 client.25531 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d20fbc310 con 0x558d20bd1390 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc00ad100 con 0x558d20bd1390 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fafb8010430 con 0x558d20bd1390 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 10 client.25531 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 10 client.25531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 20 client.25531 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20b85da0 0x558d20fbf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 4 client.25531 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.906+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d20f11df0 con 0x558d20b85da0 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.907+0000 7fafc77fe640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20b85da0 0x558d20fbf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.908+0000 7fafa77fe640 10 client.25531 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.908+0000 7fafc77fe640 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20b85da0 0x558d20fbf880 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9029aaa07e5dd350 server_cookie=ce4022dc70644d4a in_seq=0 out_seq=0 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafa77fe640 1 -- 172.21.15.28:0/3674140073 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558d20f11df0 con 0x558d20b85da0 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafa77fe640 10 client.25531 handle_command_reply: tid=0 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafd4e1c740 1 client.25531 shutdown 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 msgr2=0x558d20fadd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fafb8012770 tx=0x7fafb8012c30 comp rx=0 tx=0).stop 2024-09-21T05:47:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 shutdown_connections 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20b85da0 0x558d20fbf880 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.909+0000 7fafa77fe640 0 client.25531 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fb3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20bf2070 0x558d20fae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/3674140073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fadd80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 >> 172.21.15.28:0/3674140073 conn(0x558d20be6460 msgr2=0x558d20f96570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 shutdown_connections 2024-09-21T05:47:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 1 -- 172.21.15.28:0/3674140073 wait complete. 2024-09-21T05:47:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.910+0000 7fafd4e1c740 20 client.25531 trim_cache size 0 max 16384 2024-09-21T05:47:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.937+0000 7fafd4e1c740 1 Processor -- start 2024-09-21T05:47:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.937+0000 7fafd4e1c740 1 -- start start 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fac420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b5d90 con 0x558d20b85da0 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b78d0 con 0x558d20bb09f0 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafd4e1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b6460 con 0x558d20bd1390 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55754/0 (socket says 172.21.15.28:55754) 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.938+0000 7fafc77fe640 1 -- 172.21.15.28:0/352936593 learned_addr learned my addr 172.21.15.28:0/352936593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafcce45640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc7fff640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 -- 172.21.15.28:0/352936593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 msgr2=0x558d20fac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fac420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 -- 172.21.15.28:0/352936593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 msgr2=0x558d20fab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 -- 172.21.15.28:0/352936593 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d20f91840 con 0x558d20bb09f0 2024-09-21T05:47:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc7fff640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.939+0000 7fafc77fe640 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009850 tx=0x7fafbc03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b45cdcf3a7f1a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.940+0000 7fafadffb640 1 -- 172.21.15.28:0/352936593 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc056070 con 0x558d20bb09f0 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.940+0000 7fafadffb640 1 -- 172.21.15.28:0/352936593 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafbc032df0 con 0x558d20bb09f0 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.940+0000 7fafadffb640 1 -- 172.21.15.28:0/352936593 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc035040 con 0x558d20bb09f0 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 -- 172.21.15.28:0/352936593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 msgr2=0x558d20fa96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009850 tx=0x7fafbc03d000 comp rx=0 tx=0).stop 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 -- 172.21.15.28:0/352936593 shutdown_connections 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bd1390 0x558d20fac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bb09f0 0x558d20fa96a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.941+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/352936593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.942+0000 7fafd4e1c740 1 -- 172.21.15.28:0/352936593 >> 172.21.15.28:0/352936593 conn(0x558d20be6460 msgr2=0x558d20fa8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.942+0000 7fafd4e1c740 1 -- 172.21.15.28:0/352936593 shutdown_connections 2024-09-21T05:47:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.942+0000 7fafd4e1c740 1 -- 172.21.15.28:0/352936593 wait complete. 2024-09-21T05:47:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.942+0000 7fafd4e1c740 1 -- start start 2024-09-21T05:47:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.943+0000 7fafd4e1c740 1 Processor -- start 2024-09-21T05:47:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fb5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fb5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fb5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41658/0 (socket says 172.21.15.28:41658) 2024-09-21T05:47:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafc7fff640 1 -- 172.21.15.28:0/2674289571 learned_addr learned my addr 172.21.15.28:0/2674289571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bb09f0 0x558d20fb6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d21029240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b7200 con 0x558d20b85da0 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b6b30 con 0x558d20bd1390 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.944+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b3d70 con 0x558d20bb09f0 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafd4e1c740 10 client.0 _ll_register_callbacks cb 0x7fafb8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafc77fe640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bb09f0 0x558d20fb6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d21029240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 -- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bb09f0 msgr2=0x558d20fb6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bb09f0 0x558d20fb6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 -- 172.21.15.28:0/2674289571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 msgr2=0x558d20fb5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fb5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.945+0000 7fafcce45640 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d21029780 con 0x558d20bd1390 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafcce45640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d21029240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafc000efd0 tx=0x7fafc00adf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8977a1f17475bcdc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafa77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc001f040 con 0x558d20bd1390 2024-09-21T05:47:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fafc00b6cd0 con 0x558d20bd1390 2024-09-21T05:47:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafbc032df0 con 0x558d20bd1390 2024-09-21T05:47:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafc00e52d0 con 0x558d20bd1390 2024-09-21T05:47:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafd4e1c740 10 client.25511 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.946+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d2102a890 con 0x558d20bd1390 2024-09-21T05:47:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc00ad100 con 0x558d20bd1390 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fafc000f040 con 0x558d20bd1390 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 10 client.25511 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 10 client.25511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 20 client.25511 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20bf2070 0x558d2102cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 4 client.25511 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fafbc032df0 con 0x558d20bf2070 2024-09-21T05:47:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.947+0000 7fafc7fff640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20bf2070 0x558d2102cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.948+0000 7fafa77fe640 10 client.25511 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.948+0000 7fafc7fff640 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20bf2070 0x558d2102cc40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8f0d52cd2b6c2f7 server_cookie=46d7a38353029c21 in_seq=0 out_seq=0 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafa77fe640 1 -- 172.21.15.28:0/2674289571 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fafbc032df0 con 0x558d20bf2070 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafa77fe640 10 client.25511 handle_command_reply: tid=0 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 client.25511 shutdown 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 msgr2=0x558d21029240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d21029240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafc000efd0 tx=0x7fafc00adf20 comp rx=0 tx=0).stop 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 shutdown_connections 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558d20bf2070 0x558d2102cc40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d20bd1390 0x558d21029240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafa77fe640 0 client.25511 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558d20bb09f0 0x558d20fb6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 --2- 172.21.15.28:0/2674289571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d20b85da0 0x558d20fb5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.949+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 >> 172.21.15.28:0/2674289571 conn(0x558d20be6460 msgr2=0x558d20fac020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.950+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 shutdown_connections 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.950+0000 7fafd4e1c740 1 -- 172.21.15.28:0/2674289571 wait complete. 2024-09-21T05:47:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.950+0000 7fafd4e1c740 20 client.25511 trim_cache size 0 max 16384 2024-09-21T05:47:36.953 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 msgr2=0x7fafb00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 0x7fafb00b8c10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005f70 tx=0x7fafbc031400 comp rx=0 tx=0).stop 2024-09-21T05:47:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 msgr2=0x7fafc81a8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc81a8300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fafc000e940 tx=0x7fafc000ee00 comp rx=0 tx=0).stop 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 shutdown_connections 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8116640 0x7fafc81a8300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fafb00b6700 0x7fafb00b8c10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc810aee0 0x7fafc81a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 --2- 172.21.15.28:0/4223604652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc810a590 0x7fafc81a2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 >> 172.21.15.28:0/4223604652 conn(0x7fafc8070ec0 msgr2=0x7fafc810df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 shutdown_connections 2024-09-21T05:47:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:36.951+0000 7fafce8a8640 1 -- 172.21.15.28:0/4223604652 wait complete. 2024-09-21T05:47:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:37 smithi028 ceph-mon[31227]: pgmap v343: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T05:47:38.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:37 smithi181 ceph-mon[34819]: pgmap v343: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T05:47:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:37 smithi129 ceph-mon[34885]: pgmap v343: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T05:47:38.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:47:38.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:47:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 -- 172.21.15.28:0/56131286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4102a60 msgr2=0x7f98c4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 --2- 172.21.15.28:0/56131286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4102a60 0x7f98c4102e70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009900 tx=0x7f98b402a3f0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 -- 172.21.15.28:0/56131286 shutdown_connections 2024-09-21T05:47:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 --2- 172.21.15.28:0/56131286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4108cf0 0x7f98c410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 --2- 172.21.15.28:0/56131286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 --2- 172.21.15.28:0/56131286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4102a60 0x7f98c4102e70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 -- 172.21.15.28:0/56131286 >> 172.21.15.28:0/56131286 conn(0x7f98c4079730 msgr2=0x7f98c4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 -- 172.21.15.28:0/56131286 shutdown_connections 2024-09-21T05:47:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.606+0000 7f98c9020640 1 -- 172.21.15.28:0/56131286 wait complete. 2024-09-21T05:47:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.607+0000 7f98c9020640 1 Processor -- start 2024-09-21T05:47:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.608+0000 7f98c9020640 1 -- start start 2024-09-21T05:47:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.608+0000 7f98c9020640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 0x7f98c4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.608+0000 7f98c9020640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.608+0000 7f98c9020640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 0x7f98c419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c9020640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4112960 con 0x7f98c4108cf0 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 0x7f98c4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c9020640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41127e0 con 0x7f98c41083a0 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c9020640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4112ae0 con 0x7f98c4102a60 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 0x7f98c4199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:39874/0 (socket says 172.21.15.28:39874) 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c2d9d640 1 -- 172.21.15.28:0/3479744654 learned_addr learned my addr 172.21.15.28:0/3479744654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 0x7f98c419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c259c640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 -- 172.21.15.28:0/3479744654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 msgr2=0x7f98c4199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 0x7f98c4199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 -- 172.21.15.28:0/3479744654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 msgr2=0x7f98c41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 -- 172.21.15.28:0/3479744654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c419f200 con 0x7f98c4108cf0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c259c640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.609+0000 7f98c359e640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 0x7f98c419eb90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00da10 tx=0x7f98bc00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.610+0000 7f98abfff640 1 -- 172.21.15.28:0/3479744654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc014d80 con 0x7f98c4108cf0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.610+0000 7f98abfff640 1 -- 172.21.15.28:0/3479744654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98bc003e20 con 0x7f98c4108cf0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.610+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c419f4f0 con 0x7f98c4108cf0 2024-09-21T05:47:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.610+0000 7f98abfff640 1 -- 172.21.15.28:0/3479744654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc004450 con 0x7f98c4108cf0 2024-09-21T05:47:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.610+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c406ae60 con 0x7f98c4108cf0 2024-09-21T05:47:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 Processor -- start 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 -- start start 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x565369899970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x565369899eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56536980fe60 con 0x5653694c9490 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653697ff290 con 0x5653694d1670 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.613+0000 7f98cf594740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5653698030a0 con 0x5653694f2350 2024-09-21T05:47:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:39888/0 (socket says 172.21.15.28:39888) 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 -- 172.21.15.28:0/3794813521 learned_addr learned my addr 172.21.15.28:0/3794813521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x565369899eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x565369899970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 -- 172.21.15.28:0/3794813521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 msgr2=0x565369899eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x565369899eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 -- 172.21.15.28:0/3794813521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 msgr2=0x565369899970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x565369899970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 -- 172.21.15.28:0/3794813521 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56536955d930 con 0x5653694f2350 2024-09-21T05:47:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x565369899970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.614+0000 7f98c259c640 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f98b8013970 tx=0x7f98b800b4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b13f4ec3219d505e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.615+0000 7f98a97fa640 1 -- 172.21.15.28:0/3794813521 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8004030 con 0x5653694f2350 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.615+0000 7f98a97fa640 1 -- 172.21.15.28:0/3794813521 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b800e960 con 0x5653694f2350 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.615+0000 7f98a97fa640 1 -- 172.21.15.28:0/3794813521 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8003850 con 0x5653694f2350 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.616+0000 7f98cf594740 1 -- 172.21.15.28:0/3794813521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 msgr2=0x565369899430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.616+0000 7f98cf594740 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f98b8013970 tx=0x7f98b800b4c0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98abfff640 1 -- 172.21.15.28:0/3479744654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f98bc00bae0 con 0x7f98c4108cf0 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 -- 172.21.15.28:0/3794813521 shutdown_connections 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x565369899eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x565369899970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 --2- 172.21.15.28:0/3794813521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x565369899430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 -- 172.21.15.28:0/3794813521 >> 172.21.15.28:0/3794813521 conn(0x5653694e6740 msgr2=0x565369897320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 -- 172.21.15.28:0/3794813521 shutdown_connections 2024-09-21T05:47:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.617+0000 7f98cf594740 1 -- 172.21.15.28:0/3794813521 wait complete. 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 -- start start 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 Processor -- start 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98abfff640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 0x7f98ac0b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98abfff640 1 -- 172.21.15.28:0/3479744654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f98bc0e7a00 con 0x7f98c4108cf0 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98c259c640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 0x7f98ac0b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x5653698a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698aac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565369660890 con 0x5653694d1670 2024-09-21T05:47:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565369799d40 con 0x5653694c9490 2024-09-21T05:47:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56536989fae0 con 0x5653694f2350 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.618+0000 7f98cf594740 10 client.0 _ll_register_callbacks cb 0x5653697fdc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55794/0 (socket says 172.21.15.28:55794) 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 -- 172.21.15.28:0/344522761 learned_addr learned my addr 172.21.15.28:0/344522761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c359e640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c259c640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 0x7f98ac0b8b50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b460 tx=0x7f98b800bcb0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 -- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 msgr2=0x5653698aac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698aac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 -- 172.21.15.28:0/344522761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 msgr2=0x5653698a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x5653698a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.619+0000 7f98c2d9d640 1 -- 172.21.15.28:0/344522761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653698ab170 con 0x5653694c9490 2024-09-21T05:47:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000c00 tx=0x7f98b4031f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca56ce94e90ddc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4040070 con 0x5653694c9490 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b800e960 con 0x5653694c9490 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f98b4004240 con 0x5653694c9490 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f98b4035040 con 0x5653694c9490 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98cf594740 10 client.25520 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.620+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5653698c4d10 con 0x5653694c9490 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b800e960 con 0x5653694c9490 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f98b403c440 con 0x5653694c9490 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 10 client.25520 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 10 client.25520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 20 client.25520 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5653694b0cd0 0x5653698c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 4 client.25520 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.621+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565369811db0 con 0x5653694b0cd0 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.622+0000 7f98c259c640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5653694b0cd0 0x5653698c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.622+0000 7f98a6ff5640 10 client.25520 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.622+0000 7f98c259c640 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5653694b0cd0 0x5653698c86d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc64d9dd50c12ee server_cookie=68415a53941e89d2 in_seq=0 out_seq=0 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.623+0000 7f98a6ff5640 1 -- 172.21.15.28:0/344522761 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x565369811db0 con 0x5653694b0cd0 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.623+0000 7f98a6ff5640 10 client.25520 handle_command_reply: tid=0 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.623+0000 7f98cf594740 1 client.25520 shutdown 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 msgr2=0x5653698a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000c00 tx=0x7f98b4031f50 comp rx=0 tx=0).stop 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 shutdown_connections 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5653694b0cd0 0x5653698c86d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698aac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x5653698a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 --2- 172.21.15.28:0/344522761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x5653698a5630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 >> 172.21.15.28:0/344522761 conn(0x5653694e6740 msgr2=0x565369895850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98a6ff5640 0 client.25520 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 shutdown_connections 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 1 -- 172.21.15.28:0/344522761 wait complete. 2024-09-21T05:47:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.624+0000 7f98cf594740 20 client.25520 trim_cache size 0 max 16384 2024-09-21T05:47:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.642+0000 7f98cf594740 1 Processor -- start 2024-09-21T05:47:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.642+0000 7f98cf594740 1 -- start start 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.642+0000 7f98cf594740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x565369895850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98cf594740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98cf594740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98cf594740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ad3c0 con 0x5653694d1670 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98cf594740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41aef00 con 0x5653694c9490 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98cf594740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ada90 con 0x5653694f2350 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x565369895850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41696/0 (socket says 172.21.15.28:41696) 2024-09-21T05:47:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c259c640 1 -- 172.21.15.28:0/2304941676 learned_addr learned my addr 172.21.15.28:0/2304941676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c359e640 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.643+0000 7f98c359e640 1 -- 172.21.15.28:0/2304941676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 msgr2=0x565369895850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c359e640 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x565369895850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c359e640 1 -- 172.21.15.28:0/2304941676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 msgr2=0x565369895d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c359e640 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c359e640 1 -- 172.21.15.28:0/2304941676 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56536989bb50 con 0x5653694f2350 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c259c640 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98c359e640 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698a2620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0e5170 tx=0x7f98bc0ad700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dde9ca5a482441c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98a8ff9640 1 -- 172.21.15.28:0/2304941676 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0ad380 con 0x5653694f2350 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98a8ff9640 1 -- 172.21.15.28:0/2304941676 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98bc0adf70 con 0x5653694f2350 2024-09-21T05:47:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.644+0000 7f98a8ff9640 1 -- 172.21.15.28:0/2304941676 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0b6e00 con 0x5653694f2350 2024-09-21T05:47:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 -- 172.21.15.28:0/2304941676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 msgr2=0x5653698a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698a2620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0e5170 tx=0x7f98bc0ad700 comp rx=0 tx=0).stop 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 -- 172.21.15.28:0/2304941676 shutdown_connections 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x5653698a2620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694d1670 0x565369895d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 --2- 172.21.15.28:0/2304941676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694c9490 0x565369895850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 -- 172.21.15.28:0/2304941676 >> 172.21.15.28:0/2304941676 conn(0x5653694e6740 msgr2=0x5653698a3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 -- 172.21.15.28:0/2304941676 shutdown_connections 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.645+0000 7f98cf594740 1 -- 172.21.15.28:0/2304941676 wait complete. 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98cf594740 1 -- start start 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98cf594740 1 Processor -- start 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98cf594740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98cf594740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x5653698ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98cf594740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x56536992bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.646+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41698/0 (socket says 172.21.15.28:41698) 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98cf594740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ae830 con 0x5653694c9490 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 -- 172.21.15.28:0/2579359693 learned_addr learned my addr 172.21.15.28:0/2579359693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ae160 con 0x5653694d1670 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ab3a0 con 0x5653694f2350 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c259c640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x5653698ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c359e640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x56536992bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98cf594740 10 client.0 _ll_register_callbacks cb 0x7f98b4031be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 -- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 msgr2=0x56536992bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x56536992bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 -- 172.21.15.28:0/2579359693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 msgr2=0x5653698ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x5653698ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98c2d9d640 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56536992c600 con 0x5653694c9490 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.647+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98bc0adf70 con 0x5653694c9490 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98c2d9d640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f98b4032fd0 tx=0x7f98b403ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0ab87b378408cf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98a6ff5640 10 client.15906 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98a6ff5640 1 -- 172.21.15.28:0/2579359693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4059060 con 0x5653694c9490 2024-09-21T05:47:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98a6ff5640 1 -- 172.21.15.28:0/2579359693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f98b4004230 con 0x5653694c9490 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98a6ff5640 1 -- 172.21.15.28:0/2579359693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f98b4051070 con 0x5653694c9490 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98cf594740 10 client.15906 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.648+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=27}) -- 0x565369933990 con 0x5653694c9490 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98a6ff5640 1 -- 172.21.15.28:0/2579359693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f98b40035f0 con 0x5653694c9490 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 10 client.15906 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 20 client.15906 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x565369486080 0x565369936060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 4 client.15906 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56536980d860 con 0x565369486080 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98c259c640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x565369486080 0x565369936060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98a6ff5640 10 client.15906 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:39.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.649+0000 7f98c259c640 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x565369486080 0x565369936060 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec7794f70bcf3917 server_cookie=aa57f8d9c1c7f74a in_seq=0 out_seq=0 2024-09-21T05:47:39.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.651+0000 7f98a6ff5640 1 -- 172.21.15.28:0/2579359693 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56536980d860 con 0x565369486080 2024-09-21T05:47:39.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.651+0000 7f98a6ff5640 10 client.15906 handle_command_reply: tid=0 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.651+0000 7f98cf594740 1 client.15906 shutdown 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.651+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 msgr2=0x5653698ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.651+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f98b4032fd0 tx=0x7f98b403ccb0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 shutdown_connections 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x565369486080 0x565369936060 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98a6ff5640 0 client.15906 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5653694f2350 0x56536992bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653694d1670 0x5653698ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:47:39.655 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "72636285-d802-4799-9af4-6cabaeb119f9", 2024-09-21T05:47:39.656 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:47:39.656 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 --2- 172.21.15.28:0/2579359693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653694c9490 0x5653698ca000 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 >> 172.21.15.28:0/2579359693 conn(0x5653694e6740 msgr2=0x56536989a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 shutdown_connections 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 1 -- 172.21.15.28:0/2579359693 wait complete. 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.652+0000 7f98cf594740 20 client.15906 trim_cache size 0 max 16384 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 msgr2=0x7f98ac0b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 0x7f98ac0b8b50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b460 tx=0x7f98b800bcb0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 msgr2=0x7f98c419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 0x7f98c419eb90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00da10 tx=0x7f98bc00ded0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 shutdown_connections 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98ac0b6640 0x7f98ac0b8b50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c4108cf0 0x7f98c419eb90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c41083a0 0x7f98c41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 --2- 172.21.15.28:0/3479744654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98c4102a60 0x7f98c4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 >> 172.21.15.28:0/3479744654 conn(0x7f98c4079730 msgr2=0x7f98c41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 shutdown_connections 2024-09-21T05:47:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:39.653+0000 7f98c9020640 1 -- 172.21.15.28:0/3479744654 wait complete. 2024-09-21T05:47:40.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:39 smithi028 ceph-mon[31227]: pgmap v344: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:47:40.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:39 smithi181 ceph-mon[34819]: pgmap v344: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:47:40.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:47:40.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:39 smithi129 ceph-mon[34885]: pgmap v344: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:47:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:41.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:40 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:47:41.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:40 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:40 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:47:41.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:40 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:40 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:47:41.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:40 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:40 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:47:41.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:40 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.353+0000 7fd89490f640 1 -- 172.21.15.28:0/1014387204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901062a0 msgr2=0x7fd8901066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.353+0000 7fd89490f640 1 --2- 172.21.15.28:0/1014387204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901062a0 0x7fd8901066b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd884009900 tx=0x7fd88402a410 comp rx=0 tx=0).stop 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 -- 172.21.15.28:0/1014387204 shutdown_connections 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 --2- 172.21.15.28:0/1014387204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd890106bf0 0x7fd890110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 --2- 172.21.15.28:0/1014387204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901062a0 0x7fd8901066b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 --2- 172.21.15.28:0/1014387204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890101fc0 0x7fd8901023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 -- 172.21.15.28:0/1014387204 >> 172.21.15.28:0/1014387204 conn(0x7fd890077cc0 msgr2=0x7fd890078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 -- 172.21.15.28:0/1014387204 shutdown_connections 2024-09-21T05:47:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.354+0000 7fd89490f640 1 -- 172.21.15.28:0/1014387204 wait complete. 2024-09-21T05:47:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.355+0000 7fd89490f640 1 Processor -- start 2024-09-21T05:47:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.355+0000 7fd89490f640 1 -- start start 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8901062a0 0x7fd89019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 0x7fd89019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41722/0 (socket says 172.21.15.28:41722) 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88e59c640 1 -- 172.21.15.28:0/1187312961 learned_addr learned my addr 172.21.15.28:0/1187312961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd890113af0 con 0x7fd890101fc0 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd890113970 con 0x7fd890106bf0 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8901062a0 0x7fd89019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd890113c70 con 0x7fd8901062a0 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 0x7fd89019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.356+0000 7fd88ed9d640 1 -- 172.21.15.28:0/1187312961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8901062a0 msgr2=0x7fd89019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8901062a0 0x7fd89019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88ed9d640 1 -- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 msgr2=0x7fd89019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88ed9d640 1 -- 172.21.15.28:0/1187312961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8901a03f0 con 0x7fd890106bf0 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88e59c640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 0x7fd89019fcc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd888009fd0 tx=0x7fd88800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd87f7fe640 1 -- 172.21.15.28:0/1187312961 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88800ce20 con 0x7fd890106bf0 2024-09-21T05:47:41.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd87f7fe640 1 -- 172.21.15.28:0/1187312961 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd888005020 con 0x7fd890106bf0 2024-09-21T05:47:41.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8901a06e0 con 0x7fd890106bf0 2024-09-21T05:47:41.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd87f7fe640 1 -- 172.21.15.28:0/1187312961 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888010690 con 0x7fd890106bf0 2024-09-21T05:47:41.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.357+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8901a6e40 con 0x7fd890106bf0 2024-09-21T05:47:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.361+0000 7fd89ae83740 1 Processor -- start 2024-09-21T05:47:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.361+0000 7fd87f7fe640 1 -- 172.21.15.28:0/1187312961 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd888010830 con 0x7fd890106bf0 2024-09-21T05:47:41.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:40 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:47:41.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:40 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:40 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:47:41.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:40 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 -- start start 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4331270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d43317b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4331cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2d42a7e50 con 0x55c2d3f8a070 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c2d4297280 con 0x55c2d3f69390 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd89ae83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c2d429b090 con 0x55c2d3f611b0 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd88dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4331270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd88dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4331270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41736/0 (socket says 172.21.15.28:41736) 2024-09-21T05:47:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.362+0000 7fd88dd9b640 1 -- 172.21.15.28:0/3018117175 learned_addr learned my addr 172.21.15.28:0/3018117175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.363+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4331cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.363+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d43317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.363+0000 7fd87f7fe640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 0x7fd8780b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 0x7fd8780b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 -- 172.21.15.28:0/3018117175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 msgr2=0x55c2d4331cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4331cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 -- 172.21.15.28:0/3018117175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 msgr2=0x55c2d4331270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4331270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 -- 172.21.15.28:0/3018117175 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2d3ff5650 con 0x55c2d3f611b0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d43317b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd880010550 tx=0x7fd880010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0add875e831f62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd87cff9640 1 -- 172.21.15.28:0/3018117175 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88000e770 con 0x55c2d3f611b0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.364+0000 7fd87cff9640 1 -- 172.21.15.28:0/3018117175 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd88000edc0 con 0x55c2d3f611b0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.365+0000 7fd87cff9640 1 -- 172.21.15.28:0/3018117175 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800193d0 con 0x55c2d3f611b0 2024-09-21T05:47:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.365+0000 7fd87f7fe640 1 -- 172.21.15.28:0/1187312961 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd8880e7d80 con 0x7fd890106bf0 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 -- 172.21.15.28:0/3018117175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 msgr2=0x55c2d43317b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd88ed9d640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 0x7fd8780b8c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad890 tx=0x7fd8880aa1e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d43317b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd880010550 tx=0x7fd880010a10 comp rx=0 tx=0).stop 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 -- 172.21.15.28:0/3018117175 shutdown_connections 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4331cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d43317b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3018117175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4331270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 -- 172.21.15.28:0/3018117175 >> 172.21.15.28:0/3018117175 conn(0x55c2d3f7e460 msgr2=0x55c2d432f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 -- 172.21.15.28:0/3018117175 shutdown_connections 2024-09-21T05:47:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.366+0000 7fd89ae83740 1 -- 172.21.15.28:0/3018117175 wait complete. 2024-09-21T05:47:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.367+0000 7fd89ae83740 1 -- start start 2024-09-21T05:47:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.367+0000 7fd89ae83740 1 Processor -- start 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.367+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d4340cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.367+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4341770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd89ae83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2d40f8700 con 0x55c2d3f8a070 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd89ae83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c2d4232110 con 0x55c2d3f69390 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd89ae83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c2d4343db0 con 0x55c2d3f611b0 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd89ae83740 10 client.0 _ll_register_callbacks cb 0x55c2d424b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55832/0 (socket says 172.21.15.28:55832) 2024-09-21T05:47:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 -- 172.21.15.28:0/2515610707 learned_addr learned my addr 172.21.15.28:0/2515610707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:41.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 -- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 msgr2=0x55c2d4340cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:41.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d4340cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d4340cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 -- 172.21.15.28:0/2515610707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 msgr2=0x55c2d4341770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4341770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88e59c640 1 -- 172.21.15.28:0/2515610707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2d425fc40 con 0x55c2d3f69390 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.368+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d4340cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd88e59c640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd880006010 tx=0x7fd88000be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1181f1c98e4a545b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800114c0 con 0x55c2d3f69390 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd880011c10 con 0x55c2d3f69390 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd88000edc0 con 0x55c2d3f69390 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd880031050 con 0x55c2d3f69390 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd89ae83740 10 client.25538 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.369+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c2d434faa0 con 0x55c2d3f69390 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd88000edc0 con 0x55c2d3f69390 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd88000e570 con 0x55c2d3f69390 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 10 client.25538 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 10 client.25538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 20 client.25538 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f489f0 0x55c2d4352eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 4 client.25538 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.370+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2d42a9da0 con 0x55c2d3f489f0 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.371+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f489f0 0x55c2d4352eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.371+0000 7fd875ffb640 10 client.25538 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.371+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f489f0 0x55c2d4352eb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4420aedfeaf03795 server_cookie=36d3b033ac5c4692 in_seq=0 out_seq=0 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd875ffb640 1 -- 172.21.15.28:0/2515610707 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c2d42a9da0 con 0x55c2d3f489f0 2024-09-21T05:47:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd875ffb640 10 client.25538 handle_command_reply: tid=0 2024-09-21T05:47:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 client.25538 shutdown 2024-09-21T05:47:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 msgr2=0x55c2d4341210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd880006010 tx=0x7fd88000be70 comp rx=0 tx=0).stop 2024-09-21T05:47:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 shutdown_connections 2024-09-21T05:47:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f489f0 0x55c2d4352eb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f8a070 0x55c2d4341770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d4341210 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 --2- 172.21.15.28:0/2515610707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f611b0 0x55c2d4340cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.372+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 >> 172.21.15.28:0/2515610707 conn(0x55c2d3f7e460 msgr2=0x55c2d4332c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.373+0000 7fd875ffb640 0 client.25538 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.373+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 shutdown_connections 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.373+0000 7fd89ae83740 1 -- 172.21.15.28:0/2515610707 wait complete. 2024-09-21T05:47:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.373+0000 7fd89ae83740 20 client.25538 trim_cache size 0 max 16384 2024-09-21T05:47:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 Processor -- start 2024-09-21T05:47:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 -- start start 2024-09-21T05:47:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 0x55c2d4341190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d432efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901ad4c0 con 0x55c2d3f69390 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901af000 con 0x55c2d3f611b0 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd89ae83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901adb90 con 0x55c2d3f8a070 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd88e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41760/0 (socket says 172.21.15.28:41760) 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.408+0000 7fd88e59c640 1 -- 172.21.15.28:0/480944233 learned_addr learned my addr 172.21.15.28:0/480944233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 0x55c2d4341190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d432efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 -- 172.21.15.28:0/480944233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 msgr2=0x55c2d432efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d432efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 -- 172.21.15.28:0/480944233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 msgr2=0x55c2d4341190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 0x55c2d4341190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 -- 172.21.15.28:0/480944233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2d434d010 con 0x55c2d3f69390 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 0x55c2d4341190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd88e59c640 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd88000bf40 tx=0x7fd880010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfc46b72f1cb5e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd877fff640 1 -- 172.21.15.28:0/480944233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88003b070 con 0x55c2d3f69390 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd877fff640 1 -- 172.21.15.28:0/480944233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd880028070 con 0x55c2d3f69390 2024-09-21T05:47:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.409+0000 7fd877fff640 1 -- 172.21.15.28:0/480944233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880011440 con 0x55c2d3f69390 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 -- 172.21.15.28:0/480944233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 msgr2=0x55c2d432ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd88000bf40 tx=0x7fd880010a10 comp rx=0 tx=0).stop 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 -- 172.21.15.28:0/480944233 shutdown_connections 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d432efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f69390 0x55c2d432ea80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 --2- 172.21.15.28:0/480944233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f611b0 0x55c2d4341190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.410+0000 7fd89ae83740 1 -- 172.21.15.28:0/480944233 >> 172.21.15.28:0/480944233 conn(0x55c2d3f7e460 msgr2=0x55c2d4340a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.411+0000 7fd89ae83740 1 -- 172.21.15.28:0/480944233 shutdown_connections 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.411+0000 7fd89ae83740 1 -- 172.21.15.28:0/480944233 wait complete. 2024-09-21T05:47:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.411+0000 7fd89ae83740 1 -- start start 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.411+0000 7fd89ae83740 1 Processor -- start 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d43499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d4338390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901ae930 con 0x55c2d3f611b0 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901ae260 con 0x55c2d3f69390 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901ab4a0 con 0x55c2d3f8a070 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd89ae83740 10 client.0 _ll_register_callbacks cb 0x7fd884009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41764/0 (socket says 172.21.15.28:41764) 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 -- 172.21.15.28:0/3371161707 learned_addr learned my addr 172.21.15.28:0/3371161707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d4338390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d43499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 -- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 msgr2=0x55c2d4338390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.412+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d4338390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd88dd9b640 1 -- 172.21.15.28:0/3371161707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 msgr2=0x55c2d43499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d43499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd88dd9b640 1 -- 172.21.15.28:0/3371161707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2d4338b00 con 0x55c2d3f611b0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd88dd9b640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd8840092a0 tx=0x7fd884046f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63f43a6edeabafe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884035040 con 0x55c2d3f611b0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd8840510d0 con 0x55c2d3f611b0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd880028070 con 0x55c2d3f611b0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd8840512d0 con 0x55c2d3f611b0 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd89ae83740 10 client.15921 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.413+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c2d4346b30 con 0x55c2d3f611b0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.414+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd880028070 con 0x55c2d3f611b0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd884004870 con 0x55c2d3f611b0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 10 client.15921 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 20 client.15921 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f1dda0 0x55c2d43bb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 4 client.15921 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c2d42a5850 con 0x55c2d3f1dda0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.415+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f1dda0 0x55c2d43bb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd88e59c640 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f1dda0 0x55c2d43bb800 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b08ef74e3208164 server_cookie=95d01f254e1b968e in_seq=0 out_seq=0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd875ffb640 10 client.15921 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd875ffb640 1 -- 172.21.15.28:0/3371161707 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+942 (crc 0 0 0) 0x55c2d42a5850 con 0x55c2d3f1dda0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd875ffb640 10 client.15921 handle_command_reply: tid=0 2024-09-21T05:47:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 client.15921 shutdown 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 msgr2=0x55c2d4349410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd8840092a0 tx=0x7fd884046f90 comp rx=0 tx=0).stop 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 shutdown_connections 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c2d3f1dda0 0x55c2d43bb800 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2d3f8a070 0x55c2d4338390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c2d3f69390 0x55c2d43499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 --2- 172.21.15.28:0/3371161707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2d3f611b0 0x55c2d4349410 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 >> 172.21.15.28:0/3371161707 conn(0x55c2d3f7e460 msgr2=0x55c2d43322a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.416+0000 7fd875ffb640 0 client.15921 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 shutdown_connections 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89ae83740 1 -- 172.21.15.28:0/3371161707 wait complete. 2024-09-21T05:47:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89ae83740 20 client.15921 trim_cache size 0 max 16384 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:47:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:41.424 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 msgr2=0x7fd8780b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 0x7fd8780b8c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad890 tx=0x7fd8880aa1e0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 msgr2=0x7fd89019fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.417+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 0x7fd89019fcc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd888009fd0 tx=0x7fd88800ef10 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 shutdown_connections 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd890106bf0 0x7fd89019fcc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8901062a0 0x7fd89019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd8780b6700 0x7fd8780b8c10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 --2- 172.21.15.28:0/1187312961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd890101fc0 0x7fd89019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 >> 172.21.15.28:0/1187312961 conn(0x7fd890077cc0 msgr2=0x7fd890107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 shutdown_connections 2024-09-21T05:47:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:41.418+0000 7fd89490f640 1 -- 172.21.15.28:0/1187312961 wait complete. 2024-09-21T05:47:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:41 smithi028 ceph-mon[31227]: pgmap v345: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T05:47:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:41 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:47:42.136 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:47:42.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:47:42.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:41 smithi181 ceph-mon[34819]: pgmap v345: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T05:47:42.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:41 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:47:42.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:41 smithi129 ceph-mon[34885]: pgmap v345: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T05:47:42.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:41 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:47:42.489 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 -- 172.21.15.28:0/4226587895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 msgr2=0x7fe42c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 --2- 172.21.15.28:0/4226587895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c10f520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe424009be0 tx=0x7fe42402a6c0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 -- 172.21.15.28:0/4226587895 shutdown_connections 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 --2- 172.21.15.28:0/4226587895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c10f520 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 --2- 172.21.15.28:0/4226587895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 0x7fe42c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 --2- 172.21.15.28:0/4226587895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.991+0000 7fe432c1b640 1 -- 172.21.15.28:0/4226587895 >> 172.21.15.28:0/4226587895 conn(0x7fe42c079730 msgr2=0x7fe42c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.992+0000 7fe432c1b640 1 -- 172.21.15.28:0/4226587895 shutdown_connections 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.992+0000 7fe432c1b640 1 -- 172.21.15.28:0/4226587895 wait complete. 2024-09-21T05:47:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 Processor -- start 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 -- start start 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 0x7fe42c199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c112960 con 0x7fe42c102a60 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1127e0 con 0x7fe42c1083a0 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe432c1b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c112ae0 con 0x7fe42c108cf0 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe4309b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe4309b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41788/0 (socket says 172.21.15.28:41788) 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.993+0000 7fe4309b7640 1 -- 172.21.15.28:0/1976299796 learned_addr learned my addr 172.21.15.28:0/1976299796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4311b8640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe42bfff640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 0x7fe42c199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 -- 172.21.15.28:0/1976299796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 msgr2=0x7fe42c199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 -- 172.21.15.28:0/1976299796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 msgr2=0x7fe42c199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 0x7fe42c199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 -- 172.21.15.28:0/1976299796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c19e2e0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe4309b7640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe42000e9c0 tx=0x7fe42000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42000cde0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.994+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4200035f0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.995+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420010690 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.996+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c19e5d0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.996+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c19eae0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:42.997+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42c103ce0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.000+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe4200037a0 con 0x7fe42c102a60 2024-09-21T05:47:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.001+0000 7fe429ffb640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 0x7fe41c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.001+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe4200e79e0 con 0x7fe42c102a60 2024-09-21T05:47:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.001+0000 7fe42bfff640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 0x7fe41c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.002+0000 7fe42bfff640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 0x7fe41c0b8a70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe42c105f80 tx=0x7fe418005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.006+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4200adf50 con 0x7fe42c102a60 2024-09-21T05:47:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.235+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe42c06b300 con 0x7fe42c102a60 2024-09-21T05:47:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.236+0000 7fe429ffb640 1 -- 172.21.15.28:0/1976299796 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7fe42c06b300 con 0x7fe42c102a60 2024-09-21T05:47:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:47:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:47:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 msgr2=0x7fe41c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 0x7fe41c0b8a70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe42c105f80 tx=0x7fe418005f90 comp rx=0 tx=0).stop 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 msgr2=0x7fe42c199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe42000e9c0 tx=0x7fe42000ee80 comp rx=0 tx=0).stop 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 shutdown_connections 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe42c108cf0 0x7fe42c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c1083a0 0x7fe42c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe41c0b6560 0x7fe41c0b8a70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 --2- 172.21.15.28:0/1976299796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42c102a60 0x7fe42c199000 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:43.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 >> 172.21.15.28:0/1976299796 conn(0x7fe42c079730 msgr2=0x7fe42c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:43.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 shutdown_connections 2024-09-21T05:47:43.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:43.239+0000 7fe432c1b640 1 -- 172.21.15.28:0/1976299796 wait complete. 2024-09-21T05:47:43.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:47:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:44.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:43 smithi028 ceph-mon[31227]: pgmap v346: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T05:47:44.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:43 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1976299796' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:44.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:43 smithi181 ceph-mon[34819]: pgmap v346: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T05:47:44.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:43 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1976299796' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:43 smithi129 ceph-mon[34885]: pgmap v346: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T05:47:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:43 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1976299796' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 -- 172.21.15.28:0/3050873684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 msgr2=0x7f4238110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 --2- 172.21.15.28:0/3050873684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f4238110640 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4230009ca0 tx=0x7f423002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 -- 172.21.15.28:0/3050873684 shutdown_connections 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 --2- 172.21.15.28:0/3050873684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f4238110640 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 --2- 172.21.15.28:0/3050873684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 0x7f42381098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 --2- 172.21.15.28:0/3050873684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 0x7f4238103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 -- 172.21.15.28:0/3050873684 >> 172.21.15.28:0/3050873684 conn(0x7f42380ff610 msgr2=0x7f4238101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 -- 172.21.15.28:0/3050873684 shutdown_connections 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.947+0000 7f42404e0640 1 -- 172.21.15.28:0/3050873684 wait complete. 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 Processor -- start 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 -- start start 2024-09-21T05:47:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 0x7f423819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 0x7f423819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4238113ad0 con 0x7f4238109e10 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4238113950 con 0x7f4238103b80 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.948+0000 7f42404e0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4238113c50 con 0x7f42381094c0 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41794/0 (socket says 172.21.15.28:41794) 2024-09-21T05:47:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 -- 172.21.15.28:0/3047525761 learned_addr learned my addr 172.21.15.28:0/3047525761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423e27c640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 0x7f423819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423da7b640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 0x7f423819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 -- 172.21.15.28:0/3047525761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 msgr2=0x7f423819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 0x7f423819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 -- 172.21.15.28:0/3047525761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 msgr2=0x7f423819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 0x7f423819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 -- 172.21.15.28:0/3047525761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42381a25d0 con 0x7f4238109e10 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f423ea7d640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4230005cc0 tx=0x7f4230031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f422f7fe640 1 -- 172.21.15.28:0/3047525761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230003b70 con 0x7f4238109e10 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42381a2860 con 0x7f4238109e10 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.949+0000 7f422f7fe640 1 -- 172.21.15.28:0/3047525761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42300318f0 con 0x7f4238109e10 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.950+0000 7f422f7fe640 1 -- 172.21.15.28:0/3047525761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423002f950 con 0x7f4238109e10 2024-09-21T05:47:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.950+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42381a9020 con 0x7f4238109e10 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.952+0000 7f4246a54740 1 Processor -- start 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f422f7fe640 1 -- 172.21.15.28:0/3047525761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f423002fb50 con 0x7f4238109e10 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 -- start start 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d226a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d23120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adf9c98ea0 con 0x55adf99521b0 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55adf9c882d0 con 0x55adf995a390 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f4246a54740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55adf9c8c0e0 con 0x55adf997b070 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41810/0 (socket says 172.21.15.28:41810) 2024-09-21T05:47:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.953+0000 7f423e27c640 1 -- 172.21.15.28:0/491615457 learned_addr learned my addr 172.21.15.28:0/491615457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423da7b640 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423da7b640 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d23120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f422f7fe640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 0x7f42240b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 -- 172.21.15.28:0/491615457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 msgr2=0x55adf9d226a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d226a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 -- 172.21.15.28:0/491615457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 msgr2=0x55adf9d23120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d23120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 -- 172.21.15.28:0/491615457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf99e6650 con 0x55adf99521b0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423ea7d640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 0x7f42240b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.954+0000 7f423e27c640 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4234009cf0 tx=0x7f4234011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbd7d86b004dbc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.955+0000 7f422f7fe640 1 -- 172.21.15.28:0/3047525761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f42301087d0 con 0x7f4238109e10 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.955+0000 7f422cff9640 1 -- 172.21.15.28:0/491615457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234004230 con 0x55adf99521b0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.955+0000 7f422cff9640 1 -- 172.21.15.28:0/491615457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42340043d0 con 0x55adf99521b0 2024-09-21T05:47:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.955+0000 7f422cff9640 1 -- 172.21.15.28:0/491615457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234019820 con 0x55adf99521b0 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 -- 172.21.15.28:0/491615457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 msgr2=0x55adf9d22be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f423ea7d640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 0x7f42240b8b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f42300cb8b0 tx=0x7f42300d2040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4234009cf0 tx=0x7f4234011810 comp rx=0 tx=0).stop 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 -- 172.21.15.28:0/491615457 shutdown_connections 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d23120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d22be0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 --2- 172.21.15.28:0/491615457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d226a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 -- 172.21.15.28:0/491615457 >> 172.21.15.28:0/491615457 conn(0x55adf996f460 msgr2=0x55adf9d20590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 -- 172.21.15.28:0/491615457 shutdown_connections 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.956+0000 7f4246a54740 1 -- 172.21.15.28:0/491615457 wait complete. 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 -- start start 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 Processor -- start 2024-09-21T05:47:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d3a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adf9ae9700 con 0x55adf99521b0 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55adf9c23160 con 0x55adf995a390 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55adf9d30e80 con 0x55adf997b070 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f4246a54740 10 client.0 _ll_register_callbacks cb 0x55adf9c3c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f423da7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f423e27c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f423e27c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55902/0 (socket says 172.21.15.28:55902) 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f423da7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41826/0 (socket says 172.21.15.28:41826) 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.957+0000 7f423e27c640 1 -- 172.21.15.28:0/438234825 learned_addr learned my addr 172.21.15.28:0/438234825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423da7b640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d3a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 -- 172.21.15.28:0/438234825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 msgr2=0x55adf9d3a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d3a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 -- 172.21.15.28:0/438234825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 msgr2=0x55adf9d2dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 -- 172.21.15.28:0/438234825 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf9c50c90 con 0x55adf995a390 2024-09-21T05:47:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423da7b640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f423e27c640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4234003f50 tx=0x7f42340045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963bc67e03dc4011 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f4221ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234027070 con 0x55adf995a390 2024-09-21T05:47:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f42340035f0 con 0x55adf995a390 2024-09-21T05:47:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.958+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42340043d0 con 0x55adf995a390 2024-09-21T05:47:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4234033050 con 0x55adf995a390 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 10 client.25562 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adf9d3cbb0 con 0x55adf995a390 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42340043d0 con 0x55adf995a390 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f42340049c0 con 0x55adf995a390 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 10 client.25562 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 10 client.25562 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 20 client.25562 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf99399f0 0x55adf9d3ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 4 client.25562 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.959+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adf9c9adf0 con 0x55adf99399f0 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.960+0000 7f423da7b640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf99399f0 0x55adf9d3ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.960+0000 7f4221ffb640 10 client.25562 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.960+0000 7f423da7b640 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf99399f0 0x55adf9d3ff70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=120d9470aae2ce30 server_cookie=b5d5f30f0b3bb70b in_seq=0 out_seq=0 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4221ffb640 1 -- 172.21.15.28:0/438234825 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55adf9c9adf0 con 0x55adf99399f0 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4221ffb640 10 client.25562 handle_command_reply: tid=0 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 client.25562 shutdown 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 msgr2=0x55adf9d2e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4234003f50 tx=0x7f42340045f0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 shutdown_connections 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf99399f0 0x55adf9d3ff70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf997b070 0x55adf9d3a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d2e2b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4221ffb640 0 client.25562 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 --2- 172.21.15.28:0/438234825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d2dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 >> 172.21.15.28:0/438234825 conn(0x55adf996f460 msgr2=0x55adf9d25830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 shutdown_connections 2024-09-21T05:47:44.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 1 -- 172.21.15.28:0/438234825 wait complete. 2024-09-21T05:47:44.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.961+0000 7f4246a54740 20 client.25562 trim_cache size 0 max 16384 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 Processor -- start 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 -- start start 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf99399f0 0x55adf9d2f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 0x55adf9d20100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42381b18e0 con 0x55adf99521b0 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42381b3420 con 0x55adf99399f0 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f4246a54740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42381b1fb0 con 0x55adf995a390 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41828/0 (socket says 172.21.15.28:41828) 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f423e27c640 1 -- 172.21.15.28:0/2975561250 learned_addr learned my addr 172.21.15.28:0/2975561250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.984+0000 7f423da7b640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 0x55adf9d20100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423da7b640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf99399f0 0x55adf9d2f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 -- 172.21.15.28:0/2975561250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 msgr2=0x55adf9d20100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 0x55adf9d20100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 -- 172.21.15.28:0/2975561250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf99399f0 msgr2=0x55adf9d2f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf99399f0 0x55adf9d2f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 -- 172.21.15.28:0/2975561250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf9d24990 con 0x55adf99521b0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423da7b640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 0x55adf9d20100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f423e27c640 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4234011850 tx=0x7f4234003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21dfe0335a9a959 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f4223fff640 1 -- 172.21.15.28:0/2975561250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423403c070 con 0x55adf99521b0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f4223fff640 1 -- 172.21.15.28:0/2975561250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4234009b70 con 0x55adf99521b0 2024-09-21T05:47:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.985+0000 7f4223fff640 1 -- 172.21.15.28:0/2975561250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234027040 con 0x55adf99521b0 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 -- 172.21.15.28:0/2975561250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 msgr2=0x55adf9d1fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4234011850 tx=0x7f4234003e40 comp rx=0 tx=0).stop 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 -- 172.21.15.28:0/2975561250 shutdown_connections 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf995a390 0x55adf9d20100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d1fbc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 --2- 172.21.15.28:0/2975561250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf99399f0 0x55adf9d2f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 -- 172.21.15.28:0/2975561250 >> 172.21.15.28:0/2975561250 conn(0x55adf996f460 msgr2=0x55adf9d2f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 -- 172.21.15.28:0/2975561250 shutdown_connections 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.986+0000 7f4246a54740 1 -- 172.21.15.28:0/2975561250 wait complete. 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 -- start start 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 Processor -- start 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 0x55adf9d36820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d32210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42381b2d50 con 0x55adf99521b0 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42381b2680 con 0x55adf995a390 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42381af8c0 con 0x55adf99399f0 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.987+0000 7f4246a54740 10 client.0 _ll_register_callbacks cb 0x7f4228009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41836/0 (socket says 172.21.15.28:41836) 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 -- 172.21.15.28:0/3186165399 learned_addr learned my addr 172.21.15.28:0/3186165399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423da7b640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 0x55adf9d36820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423da7b640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d32210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 -- 172.21.15.28:0/3186165399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 msgr2=0x55adf9d36820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 0x55adf9d36820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 -- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 msgr2=0x55adf9d32210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d32210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 -- 172.21.15.28:0/3186165399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf9d328b0 con 0x55adf99521b0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423da7b640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 0x55adf9d36820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f423e27c640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4234004c60 tx=0x7f4234004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d69a987724cadac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f4221ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234044060 con 0x55adf99521b0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4234019c60 con 0x55adf99521b0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.988+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4234009b70 con 0x55adf99521b0 2024-09-21T05:47:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f423403c070 con 0x55adf99521b0 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 10 client.15945 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adf9daa620 con 0x55adf99521b0 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4234009b70 con 0x55adf99521b0 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f42340035f0 con 0x55adf99521b0 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 10 client.15945 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 10 client.15945 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 20 client.15945 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf997b070 0x55adf9dacd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 4 client.15945 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.989+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adf9c968a0 con 0x55adf997b070 2024-09-21T05:47:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f423da7b640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf997b070 0x55adf9dacd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4221ffb640 10 client.15945 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f423da7b640 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf997b070 0x55adf9dacd70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d741624bae36bb42 server_cookie=8da632d2fe0f06d5 in_seq=0 out_seq=0 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4221ffb640 1 -- 172.21.15.28:0/3186165399 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adf9c968a0 con 0x55adf997b070 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4221ffb640 10 client.15945 handle_command_reply: tid=0 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 client.15945 shutdown 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 msgr2=0x55adf9d31cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4234004c60 tx=0x7f4234004d30 comp rx=0 tx=0).stop 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 shutdown_connections 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55adf997b070 0x55adf9dacd70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55adf995a390 0x55adf9d32210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adf99521b0 0x55adf9d31cd0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.990+0000 7f4246a54740 1 --2- 172.21.15.28:0/3186165399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55adf99399f0 0x55adf9d36820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 >> 172.21.15.28:0/3186165399 conn(0x55adf996f460 msgr2=0x55adf9d236d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f4221ffb640 0 client.15945 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 shutdown_connections 2024-09-21T05:47:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f4246a54740 1 -- 172.21.15.28:0/3186165399 wait complete. 2024-09-21T05:47:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f4246a54740 20 client.15945 trim_cache size 0 max 16384 2024-09-21T05:47:44.997 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 msgr2=0x7f42240b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 0x7f42240b8b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f42300cb8b0 tx=0x7f42300d2040 comp rx=0 tx=0).stop 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.991+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 msgr2=0x7f42381a1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4230005cc0 tx=0x7f4230031700 comp rx=0 tx=0).stop 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 shutdown_connections 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f42240b6680 0x7f42240b8b90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4238109e10 0x7f42381a1ea0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42381094c0 0x7f423819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 --2- 172.21.15.28:0/3047525761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238103b80 0x7f423819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 >> 172.21.15.28:0/3047525761 conn(0x7f42380ff610 msgr2=0x7f4238108ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 shutdown_connections 2024-09-21T05:47:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:44.992+0000 7f42404e0640 1 -- 172.21.15.28:0/3047525761 wait complete. 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 -- 172.21.15.28:0/1343130103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf410dd80 msgr2=0x7f5bf4110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/1343130103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf410dd80 0x7f5bf4110260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009ca0 tx=0x7f5bec02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 -- 172.21.15.28:0/1343130103 shutdown_connections 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/1343130103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf410dd80 0x7f5bf4110260 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/1343130103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41052b0 0x7f5bf410d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/1343130103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 0x7f5bf4104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 -- 172.21.15.28:0/1343130103 >> 172.21.15.28:0/1343130103 conn(0x7f5bf407a740 msgr2=0x7f5bf407ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 -- 172.21.15.28:0/1343130103 shutdown_connections 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.783+0000 7f5bfc42e640 1 -- 172.21.15.28:0/1343130103 wait complete. 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.784+0000 7f5bfc42e640 1 Processor -- start 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.784+0000 7f5bfc42e640 1 -- start start 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.784+0000 7f5bfc42e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 0x7f5bf419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfc42e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfc42e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf410dd80 0x7f5bf419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfc42e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41139d0 con 0x7f5bf41052b0 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfc42e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf4113850 con 0x7f5bf410dd80 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfc42e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf4113b50 con 0x7f5bf4104960 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42880/0 (socket says 172.21.15.28:42880) 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 -- 172.21.15.28:0/3684979988 learned_addr learned my addr 172.21.15.28:0/3684979988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 0x7f5bf419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf410dd80 0x7f5bf419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 -- 172.21.15.28:0/3684979988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 msgr2=0x7f5bf419a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 0x7f5bf419a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 -- 172.21.15.28:0/3684979988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf410dd80 msgr2=0x7f5bf419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf410dd80 0x7f5bf419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 -- 172.21.15.28:0/3684979988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf41a03a0 con 0x7f5bf41052b0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e9f0 tx=0x7f5be400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.785+0000 7f5beb7fe640 1 -- 172.21.15.28:0/3684979988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be400ccc0 con 0x7f5bf41052b0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.786+0000 7f5beb7fe640 1 -- 172.21.15.28:0/3684979988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be40108e0 con 0x7f5bf41052b0 2024-09-21T05:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.786+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf41a0690 con 0x7f5bf41052b0 2024-09-21T05:47:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.786+0000 7f5beb7fe640 1 -- 172.21.15.28:0/3684979988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4018860 con 0x7f5bf41052b0 2024-09-21T05:47:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.786+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf41a6e20 con 0x7f5bf41052b0 2024-09-21T05:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 Processor -- start 2024-09-21T05:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 -- start start 2024-09-21T05:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5beb7fe640 1 -- 172.21.15.28:0/3684979988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5be4010430 con 0x7f5bf41052b0 2024-09-21T05:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e9167ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167abef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.789+0000 7f5c03476740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e9167206f0 con 0x55e9163e54d0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5c03476740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e91670fb20 con 0x55e9163dd2f0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5c03476740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e916713930 con 0x55e9164061b0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42888/0 (socket says 172.21.15.28:42888) 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 -- 172.21.15.28:0/1268025166 learned_addr learned my addr 172.21.15.28:0/1268025166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e9167ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 -- 172.21.15.28:0/1268025166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 msgr2=0x55e9167ab9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e9167ab9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 -- 172.21.15.28:0/1268025166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 msgr2=0x55e9167abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 -- 172.21.15.28:0/1268025166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9166d5530 con 0x55e9163e54d0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5be40e5010 tx=0x7f5be40ef690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42969d6752167fac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.790+0000 7f5beb7fe640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 0x7f5be00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.791+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 0x7f5be00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.791+0000 7f5be8bfe640 1 -- 172.21.15.28:0/1268025166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40f6070 con 0x55e9163e54d0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.791+0000 7f5beb7fe640 1 -- 172.21.15.28:0/3684979988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5be40e7930 con 0x7f5bf41052b0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.791+0000 7f5be8bfe640 1 -- 172.21.15.28:0/1268025166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be4010720 con 0x55e9163e54d0 2024-09-21T05:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.791+0000 7f5be8bfe640 1 -- 172.21.15.28:0/1268025166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40a9df0 con 0x55e9163e54d0 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- 172.21.15.28:0/1268025166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 msgr2=0x55e9167ac430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5be40e5010 tx=0x7f5be40ef690 comp rx=0 tx=0).stop 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 0x7f5be00b8c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005e30 tx=0x7f5bec005c30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- 172.21.15.28:0/1268025166 shutdown_connections 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ac430 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 --2- 172.21.15.28:0/1268025166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e9167ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- 172.21.15.28:0/1268025166 >> 172.21.15.28:0/1268025166 conn(0x55e9163fa300 msgr2=0x55e9167a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- 172.21.15.28:0/1268025166 shutdown_connections 2024-09-21T05:47:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- 172.21.15.28:0/1268025166 wait complete. 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 -- start start 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.792+0000 7f5c03476740 1 Processor -- start 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163e54d0 0x55e9167b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167c3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e916647ab0 con 0x55e9163dd2f0 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e9166b5730 con 0x55e9164061b0 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e9167ba130 con 0x55e9163e54d0 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5c03476740 10 client.0 _ll_register_callbacks cb 0x55e9167096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42894/0 (socket says 172.21.15.28:42894) 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 -- 172.21.15.28:0/1755789587 learned_addr learned my addr 172.21.15.28:0/1755789587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163e54d0 0x55e9167b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 -- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163e54d0 msgr2=0x55e9167b75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163e54d0 0x55e9167b75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 -- 172.21.15.28:0/1755789587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 msgr2=0x55e9167c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.793+0000 7f5bfa1ca640 1 -- 172.21.15.28:0/1755789587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e916471790 con 0x55e9163dd2f0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000bfc0 tx=0x7f5bf00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfee0f30cc99493a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0004020 con 0x55e9163dd2f0 2024-09-21T05:47:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5bf00041c0 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e916722640 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5bf00043c0 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 10 client.15957 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9164ea620 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e916722640 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5bf0021020 con 0x55e9163dd2f0 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 10 client.15957 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.794+0000 7f5c03476740 20 client.15957 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e9163c4d80 0x55e9167c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5c03476740 4 client.15957 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e91671e0f0 con 0x55e9163c4d80 2024-09-21T05:47:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e9163c4d80 0x55e9167c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5bddffb640 10 client.15957 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.795+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e9163c4d80 0x55e9167c9230 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f1a379868077d23 server_cookie=a9dc06f5b12c5ee3 in_seq=0 out_seq=0 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5bddffb640 1 -- 172.21.15.28:0/1755789587 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e91671e0f0 con 0x55e9163c4d80 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5bddffb640 10 client.15957 handle_command_reply: tid=0 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 client.15957 shutdown 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 msgr2=0x55e9167b70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000bfc0 tx=0x7f5bf00118d0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 shutdown_connections 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e9163c4d80 0x55e9167c9230 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163e54d0 0x55e9167b75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5bddffb640 0 client.15957 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 --2- 172.21.15.28:0/1755789587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163dd2f0 0x55e9167b70b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.796+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 >> 172.21.15.28:0/1755789587 conn(0x55e9163fa300 msgr2=0x55e9167ac9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.797+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 shutdown_connections 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.797+0000 7f5c03476740 1 -- 172.21.15.28:0/1755789587 wait complete. 2024-09-21T05:47:45.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.797+0000 7f5c03476740 20 client.15957 trim_cache size 0 max 16384 2024-09-21T05:47:45.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.826+0000 7f5c03476740 1 Processor -- start 2024-09-21T05:47:45.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 -- start start 2024-09-21T05:47:45.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163dd2f0 0x55e9167b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41ad5e0 con 0x55e9163e54d0 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41af120 con 0x55e9164061b0 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5c03476740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41adcb0 con 0x55e9163dd2f0 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42902/0 (socket says 172.21.15.28:42902) 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5bf99c9640 1 -- 172.21.15.28:0/124956483 learned_addr learned my addr 172.21.15.28:0/124956483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163dd2f0 0x55e9167b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.827+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 -- 172.21.15.28:0/124956483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163dd2f0 msgr2=0x55e9167b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163dd2f0 0x55e9167b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 -- 172.21.15.28:0/124956483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 msgr2=0x55e9167b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167b8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 -- 172.21.15.28:0/124956483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9167adcb0 con 0x55e9163e54d0 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e9c0 tx=0x7f5be40eaf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03920102bee3693 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bdffff640 1 -- 172.21.15.28:0/124956483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40acf10 con 0x55e9163e54d0 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bdffff640 1 -- 172.21.15.28:0/124956483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be40efe90 con 0x55e9163e54d0 2024-09-21T05:47:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.828+0000 7f5bdffff640 1 -- 172.21.15.28:0/124956483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40ad0f0 con 0x55e9163e54d0 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- 172.21.15.28:0/124956483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 msgr2=0x55e9167ad3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e9c0 tx=0x7f5be40eaf90 comp rx=0 tx=0).stop 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- 172.21.15.28:0/124956483 shutdown_connections 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9164061b0 0x55e9167b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167ad3a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 --2- 172.21.15.28:0/124956483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9163dd2f0 0x55e9167b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- 172.21.15.28:0/124956483 >> 172.21.15.28:0/124956483 conn(0x55e9163fa300 msgr2=0x55e9167b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- 172.21.15.28:0/124956483 shutdown_connections 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- 172.21.15.28:0/124956483 wait complete. 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.829+0000 7f5c03476740 1 -- start start 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 Processor -- start 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167bab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e916831c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41aea50 con 0x55e9163e54d0 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41ae380 con 0x55e9163dd2f0 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf41ab5e0 con 0x55e9164061b0 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5c03476740 10 client.0 _ll_register_callbacks cb 0x7f5bf000bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42918/0 (socket says 172.21.15.28:42918) 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 -- 172.21.15.28:0/41014108 learned_addr learned my addr 172.21.15.28:0/41014108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bfa9cb640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e916831c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167bab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 -- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 msgr2=0x55e916831c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e916831c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 -- 172.21.15.28:0/41014108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 msgr2=0x55e9167bab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167bab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.830+0000 7f5bf99c9640 1 -- 172.21.15.28:0/41014108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e916832160 con 0x55e9163e54d0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bf99c9640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5be40ed210 tx=0x7f5be400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c543d5a0bbf8b32 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40ad570 con 0x55e9163e54d0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5be40e5320 con 0x55e9163e54d0 2024-09-21T05:47:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5be40efe90 con 0x55e9163e54d0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5be40a9640 con 0x55e9163e54d0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 10 client.15963 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e916833370 con 0x55e9163e54d0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be40efe90 con 0x55e9163e54d0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5be40a9a10 con 0x55e9163e54d0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 10 client.15963 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 10 client.15963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 20 client.15963 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e91639a1c0 0x55e9168355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 4 client.15963 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.831+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e91671b520 con 0x55e91639a1c0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.832+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e91639a1c0 0x55e9168355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.832+0000 7f5bfa1ca640 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e91639a1c0 0x55e9168355a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b898303b3caf6ff server_cookie=1ce4ceaa6fe55c13 in_seq=0 out_seq=0 2024-09-21T05:47:45.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.832+0000 7f5bddffb640 10 client.15963 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5bddffb640 1 -- 172.21.15.28:0/41014108 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e91671b520 con 0x55e91639a1c0 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5bddffb640 10 client.15963 handle_command_reply: tid=0 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 client.15963 shutdown 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 msgr2=0x55e9167bb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5be40ed210 tx=0x7f5be400ce60 comp rx=0 tx=0).stop 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 shutdown_connections 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e91639a1c0 0x55e9168355a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e9164061b0 0x55e916831c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9163e54d0 0x55e9167bb0c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 --2- 172.21.15.28:0/41014108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e9163dd2f0 0x55e9167bab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5bddffb640 0 client.15963 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 >> 172.21.15.28:0/41014108 conn(0x55e9163fa300 msgr2=0x55e9167b8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 shutdown_connections 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 1 -- 172.21.15.28:0/41014108 wait complete. 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.838+0000 7f5c03476740 20 client.15963 trim_cache size 0 max 16384 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:47:45.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 msgr2=0x7f5be00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 0x7f5be00b8c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005e30 tx=0x7f5bec005c30 comp rx=0 tx=0).stop 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 msgr2=0x7f5bf419aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e9f0 tx=0x7f5be400eeb0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 shutdown_connections 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf410dd80 0x7f5bf419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5be00b6700 0x7f5be00b8c10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bf41052b0 0x7f5bf419aa10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 --2- 172.21.15.28:0/3684979988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bf4104960 0x7f5bf419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 >> 172.21.15.28:0/3684979988 conn(0x7f5bf407a740 msgr2=0x7f5bf4107840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 shutdown_connections 2024-09-21T05:47:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:45.839+0000 7f5bfc42e640 1 -- 172.21.15.28:0/3684979988 wait complete. 2024-09-21T05:47:45.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:47:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:46.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:45 smithi181 ceph-mon[34819]: pgmap v347: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:46.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:45 smithi028 ceph-mon[31227]: pgmap v347: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:45 smithi129 ceph-mon[34885]: pgmap v347: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 -- 172.21.15.28:0/545725182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c10de40 msgr2=0x7ffa8c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 --2- 172.21.15.28:0/545725182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c10de40 0x7ffa8c110330 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009ca0 tx=0x7ffa8402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 -- 172.21.15.28:0/545725182 shutdown_connections 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 --2- 172.21.15.28:0/545725182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c10de40 0x7ffa8c110330 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 --2- 172.21.15.28:0/545725182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c105330 0x7ffa8c10d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 --2- 172.21.15.28:0/545725182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c1049e0 0x7ffa8c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 -- 172.21.15.28:0/545725182 >> 172.21.15.28:0/545725182 conn(0x7ffa8c079730 msgr2=0x7ffa8c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 -- 172.21.15.28:0/545725182 shutdown_connections 2024-09-21T05:47:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.794+0000 7ffa918cd640 1 -- 172.21.15.28:0/545725182 wait complete. 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 Processor -- start 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 -- start start 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c1049e0 0x7ffa8c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c10de40 0x7ffa8c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c113aa0 con 0x7ffa8c105330 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c113920 con 0x7ffa8c10de40 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa918cd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c113c20 con 0x7ffa8c1049e0 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42930/0 (socket says 172.21.15.28:42930) 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.795+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/851646147 learned_addr learned my addr 172.21.15.28:0/851646147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c10de40 0x7ffa8c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8affd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c1049e0 0x7ffa8c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/851646147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c1049e0 msgr2=0x7ffa8c19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c1049e0 0x7ffa8c19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/851646147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c10de40 msgr2=0x7ffa8c19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c10de40 0x7ffa8c19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/851646147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8c1a03c0 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d740 tx=0x7ffa8000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa908cb640 1 -- 172.21.15.28:0/851646147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80017070 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8c1a06b0 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa908cb640 1 -- 172.21.15.28:0/851646147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa8000de50 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa908cb640 1 -- 172.21.15.28:0/851646147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000f620 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.796+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8c102600 con 0x7ffa8c105330 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 Processor -- start 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 -- start start 2024-09-21T05:47:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb613b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb613b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa908cb640 1 -- 172.21.15.28:0/851646147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ffa8001a070 con 0x7ffa8c105330 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555eb60b1ea0 con 0x555eb5d73390 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555eb60a12d0 con 0x555eb5d94070 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa97e41740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555eb60a50e0 con 0x555eb5d6b1b0 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.799+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42940/0 (socket says 172.21.15.28:42940) 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2489900009 learned_addr learned my addr 172.21.15.28:0/2489900009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb613b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb613b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2489900009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 msgr2=0x555eb613b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb613b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2489900009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 msgr2=0x555eb613b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb613b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2489900009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb5dff650 con 0x555eb5d73390 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffa800e4810 tx=0x7ffa800ef910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c4f8856a1dea30 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa71ffb640 1 -- 172.21.15.28:0/2489900009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800fa070 con 0x555eb5d73390 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa71ffb640 1 -- 172.21.15.28:0/2489900009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa800e7a60 con 0x555eb5d73390 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.800+0000 7ffa71ffb640 1 -- 172.21.15.28:0/2489900009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800f5b00 con 0x555eb5d73390 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa908cb640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 0x7ffa740b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 0x7ffa740b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa908cb640 1 -- 172.21.15.28:0/851646147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffa800e7130 con 0x7ffa8c105330 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 -- 172.21.15.28:0/2489900009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 msgr2=0x555eb613bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 0x7ffa740b8b90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005e30 tx=0x7ffa84005c30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffa800e4810 tx=0x7ffa800ef910 comp rx=0 tx=0).stop 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 -- 172.21.15.28:0/2489900009 shutdown_connections 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb613bd40 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb613b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2489900009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb613b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.801+0000 7ffa97e41740 1 -- 172.21.15.28:0/2489900009 >> 172.21.15.28:0/2489900009 conn(0x555eb5d88460 msgr2=0x555eb61391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 -- 172.21.15.28:0/2489900009 shutdown_connections 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 -- 172.21.15.28:0/2489900009 wait complete. 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 -- start start 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 Processor -- start 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 0x555eb6146e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.802+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa97e41740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555eb5f02700 con 0x555eb5d6b1b0 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa97e41740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555eb603c160 con 0x555eb5d73390 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa97e41740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555eb6149aa0 con 0x555eb5d94070 2024-09-21T05:47:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa97e41740 10 client.0 _ll_register_callbacks cb 0x555eb60559e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42946/0 (socket says 172.21.15.28:42946) 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 -- 172.21.15.28:0/2672809429 learned_addr learned my addr 172.21.15.28:0/2672809429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 0x555eb6146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 -- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 msgr2=0x555eb6152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 -- 172.21.15.28:0/2672809429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 msgr2=0x555eb6146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 0x555eb6146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 -- 172.21.15.28:0/2672809429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb6069c90 con 0x555eb5d6b1b0 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 0x555eb6146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:46.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009830 tx=0x7ffa7c009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf19ae3361938699 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa6affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.803+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c008c20 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa800e7a60 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffa7c008dc0 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffa7c029050 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 10 client.15975 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555eb6155830 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa800e7a60 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffa7c01a500 con 0x555eb5d6b1b0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 10 client.15975 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 10 client.15975 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 20 client.15975 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d529f0 0x555eb6158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 4 client.15975 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.804+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555eb60b3df0 con 0x555eb5d529f0 2024-09-21T05:47:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.805+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d529f0 0x555eb6158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.805+0000 7ffa6affd640 10 client.15975 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.805+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d529f0 0x555eb6158e80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e0a9a2febf7511a6 server_cookie=d67a9463b9617d66 in_seq=0 out_seq=0 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa6affd640 1 -- 172.21.15.28:0/2672809429 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555eb60b3df0 con 0x555eb5d529f0 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa6affd640 10 client.15975 handle_command_reply: tid=0 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 client.15975 shutdown 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 msgr2=0x555eb6146940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009830 tx=0x7ffa7c009cf0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 shutdown_connections 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d529f0 0x555eb6158e80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d73390 0x555eb6146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2672809429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d6b1b0 0x555eb6146940 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 >> 172.21.15.28:0/2672809429 conn(0x555eb5d88460 msgr2=0x555eb613e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa6affd640 0 client.15975 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 shutdown_connections 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.806+0000 7ffa97e41740 1 -- 172.21.15.28:0/2672809429 wait complete. 2024-09-21T05:47:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.807+0000 7ffa97e41740 20 client.15975 trim_cache size 0 max 16384 2024-09-21T05:47:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.834+0000 7ffa97e41740 1 Processor -- start 2024-09-21T05:47:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.834+0000 7ffa97e41740 1 -- start start 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d6b1b0 0x555eb6146e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6143b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1ad570 con 0x555eb5d73390 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1af0b0 con 0x555eb5d6b1b0 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa97e41740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1adc40 con 0x555eb5d94070 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42950/0 (socket says 172.21.15.28:42950) 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/1022697288 learned_addr learned my addr 172.21.15.28:0/1022697288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8affd640 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d6b1b0 0x555eb6146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/1022697288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 msgr2=0x555eb6143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/1022697288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d6b1b0 msgr2=0x555eb6146e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d6b1b0 0x555eb6146e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/1022697288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb5d52c20 con 0x555eb5d73390 2024-09-21T05:47:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.835+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffa800f0010 tx=0x7ffa800f56c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca1e6b9bfdf55cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.836+0000 7ffa717fa640 1 -- 172.21.15.28:0/1022697288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80021070 con 0x555eb5d73390 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.836+0000 7ffa717fa640 1 -- 172.21.15.28:0/1022697288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa800acab0 con 0x555eb5d73390 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.836+0000 7ffa717fa640 1 -- 172.21.15.28:0/1022697288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800ac8f0 con 0x555eb5d73390 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- 172.21.15.28:0/1022697288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 msgr2=0x555eb6152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffa800f0010 tx=0x7ffa800f56c0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- 172.21.15.28:0/1022697288 shutdown_connections 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d94070 0x555eb6143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb6152cf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 --2- 172.21.15.28:0/1022697288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d6b1b0 0x555eb6146e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- 172.21.15.28:0/1022697288 >> 172.21.15.28:0/1022697288 conn(0x555eb5d88460 msgr2=0x555eb61466f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- 172.21.15.28:0/1022697288 shutdown_connections 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- 172.21.15.28:0/1022697288 wait complete. 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 -- start start 2024-09-21T05:47:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.837+0000 7ffa97e41740 1 Processor -- start 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb614f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb614af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1ae9e0 con 0x555eb5d73390 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1ae310 con 0x555eb5d94070 2024-09-21T05:47:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8c1ab550 con 0x555eb5d6b1b0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa97e41740 10 client.0 _ll_register_callbacks cb 0x7ffa7c009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42952/0 (socket says 172.21.15.28:42952) 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2686874571 learned_addr learned my addr 172.21.15.28:0/2686874571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb614f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8b7fe640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb614af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 msgr2=0x555eb614f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb614f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2686874571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 msgr2=0x555eb614af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb614af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.838+0000 7ffa8a7fc640 1 -- 172.21.15.28:0/2686874571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb614b700 con 0x555eb5d73390 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa8a7fc640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffa800ef3d0 tx=0x7ffa80000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91aab3abf54b54ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa6affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80021030 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffa800981b0 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa800acab0 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffa800ac710 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa97e41740 10 client.15981 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555eb61c37a0 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.839+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa800acab0 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffa800a9210 con 0x555eb5d73390 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 10 client.15981 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 10 client.15981 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 20 client.15981 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d27da0 0x555eb61c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 4 client.15981 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555eb60af8a0 con 0x555eb5d27da0 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d27da0 0x555eb61c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa6affd640 10 client.15981 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.840+0000 7ffa8affd640 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d27da0 0x555eb61c5a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=339b64f76f9a68 server_cookie=c2bd0d2b086627ac in_seq=0 out_seq=0 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa6affd640 1 -- 172.21.15.28:0/2686874571 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555eb60af8a0 con 0x555eb5d27da0 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa6affd640 10 client.15981 handle_command_reply: tid=0 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 client.15981 shutdown 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 msgr2=0x555eb614aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffa800ef3d0 tx=0x7ffa80000bc0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 shutdown_connections 2024-09-21T05:47:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555eb5d27da0 0x555eb61c5a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555eb5d94070 0x555eb614af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa6affd640 0 client.15981 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555eb5d73390 0x555eb614aa10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 --2- 172.21.15.28:0/2686874571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555eb5d6b1b0 0x555eb614f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 >> 172.21.15.28:0/2686874571 conn(0x555eb5d88460 msgr2=0x555eb613c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 shutdown_connections 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 1 -- 172.21.15.28:0/2686874571 wait complete. 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.841+0000 7ffa97e41740 20 client.15981 trim_cache size 0 max 16384 2024-09-21T05:47:46.847 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 msgr2=0x7ffa740b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 0x7ffa740b8b90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005e30 tx=0x7ffa84005c30 comp rx=0 tx=0).stop 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 msgr2=0x7ffa8c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d740 tx=0x7ffa8000dc00 comp rx=0 tx=0).stop 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 shutdown_connections 2024-09-21T05:47:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa8c10de40 0x7ffa8c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffa740b6680 0x7ffa740b8b90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa8c105330 0x7ffa8c19a9f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 --2- 172.21.15.28:0/851646147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa8c1049e0 0x7ffa8c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 >> 172.21.15.28:0/851646147 conn(0x7ffa8c079730 msgr2=0x7ffa8c1078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 shutdown_connections 2024-09-21T05:47:46.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:46.842+0000 7ffa918cd640 1 -- 172.21.15.28:0/851646147 wait complete. 2024-09-21T05:47:46.880 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 88% (11178/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-21T05:47:47.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:47:47.819 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T05:47:47.819 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-09-21T05:47:47.820 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.821 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T05:47:47.821 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-21T05:47:47.821 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T05:47:47.825 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T05:47:47.855 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T05:47:47.857 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T05:47:47.858 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T05:47:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:48.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: pgmap v348: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:48.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:47 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: pgmap v348: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:47 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: pgmap v348: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:47 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 -- 172.21.15.28:0/598416327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc80102a80 msgr2=0x7efc80102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 --2- 172.21.15.28:0/598416327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc80102a80 0x7efc80102e90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efc70009900 tx=0x7efc7002a410 comp rx=0 tx=0).stop 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 -- 172.21.15.28:0/598416327 shutdown_connections 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 --2- 172.21.15.28:0/598416327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 0x7efc8010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 --2- 172.21.15.28:0/598416327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc801083c0 0x7efc801087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 --2- 172.21.15.28:0/598416327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc80102a80 0x7efc80102e90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 -- 172.21.15.28:0/598416327 >> 172.21.15.28:0/598416327 conn(0x7efc80079740 msgr2=0x7efc80079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 -- 172.21.15.28:0/598416327 shutdown_connections 2024-09-21T05:47:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 -- 172.21.15.28:0/598416327 wait complete. 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.514+0000 7efc8624a640 1 Processor -- start 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 -- start start 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc80102a80 0x7efc8019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 0x7efc8019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc80113aa0 con 0x7efc801083c0 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc80113920 con 0x7efc80108d10 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc8624a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc80113c20 con 0x7efc80102a80 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42966/0 (socket says 172.21.15.28:42966) 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 -- 172.21.15.28:0/2272881063 learned_addr learned my addr 172.21.15.28:0/2272881063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc80102a80 0x7efc8019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7ffff640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 0x7efc8019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 -- 172.21.15.28:0/2272881063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc80102a80 msgr2=0x7efc8019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc80102a80 0x7efc8019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 -- 172.21.15.28:0/2272881063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 msgr2=0x7efc8019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 0x7efc8019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.515+0000 7efc7effd640 1 -- 172.21.15.28:0/2272881063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc801a0420 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc7effd640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e9c0 tx=0x7efc7400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc7cff9640 1 -- 172.21.15.28:0/2272881063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7400cd80 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc7cff9640 1 -- 172.21.15.28:0/2272881063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc740054d0 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc801a0710 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc7cff9640 1 -- 172.21.15.28:0/2272881063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74010430 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.516+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc801a6e70 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.518+0000 7efc8c7be740 1 Processor -- start 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.518+0000 7efc8c7be740 1 -- start start 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7cff9640 1 -- 172.21.15.28:0/2272881063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7efc74004fd0 con 0x7efc801083c0 2024-09-21T05:47:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9f5070 0x55809fd9c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 0x55809fd9cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55809fd12ea0 con 0x55809f9d4390 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55809fd022d0 con 0x55809f9f5070 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc8c7be740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55809fd060e0 con 0x55809f9cc1b0 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 0x55809fd9cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9f5070 0x55809fd9c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42976/0 (socket says 172.21.15.28:42976) 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 -- 172.21.15.28:0/1183965290 learned_addr learned my addr 172.21.15.28:0/1183965290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 0x55809fd9cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51688/0 (socket says 172.21.15.28:51688) 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 -- 172.21.15.28:0/1183965290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 msgr2=0x55809fd9cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 0x55809fd9cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 -- 172.21.15.28:0/1183965290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9f5070 msgr2=0x55809fd9c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9f5070 0x55809fd9c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.519+0000 7efc7effd640 1 -- 172.21.15.28:0/1183965290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55809fa60650 con 0x55809f9d4390 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc7effd640 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efc740e52a0 tx=0x7efc740ebd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1748f2f48f69b1b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc7cff9640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 0x7efc680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc65ffb640 1 -- 172.21.15.28:0/1183965290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc740f0cf0 con 0x55809f9d4390 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc65ffb640 1 -- 172.21.15.28:0/1183965290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc740ad540 con 0x55809f9d4390 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc65ffb640 1 -- 172.21.15.28:0/1183965290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc740ad380 con 0x55809f9d4390 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc7ffff640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 0x7efc680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.520+0000 7efc7cff9640 1 -- 172.21.15.28:0/2272881063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc740e7bc0 con 0x7efc801083c0 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- 172.21.15.28:0/1183965290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 msgr2=0x55809fd9d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc7ffff640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 0x7efc680b8b90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efc78005fd0 tx=0x7efc78009740 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efc740e52a0 tx=0x7efc740ebd00 comp rx=0 tx=0).stop 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- 172.21.15.28:0/1183965290 shutdown_connections 2024-09-21T05:47:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fd9d120 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9cc1b0 0x55809fd9cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 --2- 172.21.15.28:0/1183965290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9f5070 0x55809fd9c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- 172.21.15.28:0/1183965290 >> 172.21.15.28:0/1183965290 conn(0x55809f9e9460 msgr2=0x55809fd9a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- 172.21.15.28:0/1183965290 shutdown_connections 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- 172.21.15.28:0/1183965290 wait complete. 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 -- start start 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.521+0000 7efc8c7be740 1 Processor -- start 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fda7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9f5070 0x55809fdb4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55809fb63700 con 0x55809f9d4390 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55809fc9d160 con 0x55809f9cc1b0 2024-09-21T05:47:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55809fdaae80 con 0x55809f9f5070 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc8c7be740 10 client.0 _ll_register_callbacks cb 0x55809fcb69e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42984/0 (socket says 172.21.15.28:42984) 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 -- 172.21.15.28:0/3404263103 learned_addr learned my addr 172.21.15.28:0/3404263103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fda7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7ffff640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9f5070 0x55809fdb4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 -- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9f5070 msgr2=0x55809fdb4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9f5070 0x55809fdb4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 -- 172.21.15.28:0/3404263103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 msgr2=0x55809fda7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fda7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 -- 172.21.15.28:0/3404263103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55809fccac90 con 0x55809f9d4390 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc7effd640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efc740ad9c0 tx=0x7efc7400efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b9a5c0e97e0ed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.522+0000 7efc5effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc740dd630 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc740b6fc0 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc740ad540 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc74096020 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 10 client.15993 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55809fdb6ba0 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc740ad540 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efc740b20d0 con 0x55809f9d4390 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 10 client.15993 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 10 client.15993 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 20 client.15993 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9b39f0 0x55809fdb9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 4 client.15993 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.523+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55809fd14df0 con 0x55809f9b39f0 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.524+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9b39f0 0x55809fdb9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.524+0000 7efc5effd640 10 client.15993 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.524+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9b39f0 0x55809fdb9f00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce726698b79515d5 server_cookie=c898dc2034d8e97 in_seq=0 out_seq=0 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc5effd640 1 -- 172.21.15.28:0/3404263103 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55809fd14df0 con 0x55809f9b39f0 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc5effd640 10 client.15993 handle_command_reply: tid=0 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 client.15993 shutdown 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 msgr2=0x55809fda82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efc740ad9c0 tx=0x7efc7400efd0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 shutdown_connections 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9b39f0 0x55809fdb9f00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc5effd640 0 client.15993 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9f5070 0x55809fdb4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9d4390 0x55809fda82b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3404263103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fda7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 >> 172.21.15.28:0/3404263103 conn(0x55809f9e9460 msgr2=0x55809fd9d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 shutdown_connections 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.525+0000 7efc8c7be740 1 -- 172.21.15.28:0/3404263103 wait complete. 2024-09-21T05:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.526+0000 7efc8c7be740 20 client.15993 trim_cache size 0 max 16384 2024-09-21T05:47:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.539+0000 7efc8c7be740 1 Processor -- start 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 -- start start 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fd99bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fd9a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc801af750 con 0x55809f9b39f0 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc801b1290 con 0x55809f9cc1b0 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc8c7be740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc801afe20 con 0x55809f9d4390 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42998/0 (socket says 172.21.15.28:42998) 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 -- 172.21.15.28:0/3724028725 learned_addr learned my addr 172.21.15.28:0/3724028725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7effd640 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fd99bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7ffff640 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fd9a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 -- 172.21.15.28:0/3724028725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 msgr2=0x55809fd9a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fd9a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 -- 172.21.15.28:0/3724028725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 msgr2=0x55809fd99bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fd99bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.540+0000 7efc7f7fe640 1 -- 172.21.15.28:0/3724028725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55809fdc01f0 con 0x55809f9b39f0 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.541+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efc7002a930 tx=0x7efc70005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b271f7c5e6e3a2e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.541+0000 7efc657fa640 1 -- 172.21.15.28:0/3724028725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70035040 con 0x55809f9b39f0 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.541+0000 7efc657fa640 1 -- 172.21.15.28:0/3724028725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc700054f0 con 0x55809f9b39f0 2024-09-21T05:47:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.541+0000 7efc657fa640 1 -- 172.21.15.28:0/3724028725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700035f0 con 0x55809f9b39f0 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- 172.21.15.28:0/3724028725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 msgr2=0x55809fd9dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efc7002a930 tx=0x7efc70005db0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- 172.21.15.28:0/3724028725 shutdown_connections 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fd9a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9cc1b0 0x55809fd99bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 --2- 172.21.15.28:0/3724028725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9b39f0 0x55809fd9dde0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- 172.21.15.28:0/3724028725 >> 172.21.15.28:0/3724028725 conn(0x55809f9e9460 msgr2=0x55809fd9d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- 172.21.15.28:0/3724028725 shutdown_connections 2024-09-21T05:47:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- 172.21.15.28:0/3724028725 wait complete. 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 -- start start 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.542+0000 7efc8c7be740 1 Processor -- start 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 0x55809fdbae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 0x55809fdbb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fdbb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc801b0bc0 con 0x55809f9cc1b0 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc801b04f0 con 0x55809f9b39f0 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc801ad730 con 0x55809f9d4390 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 0x55809fdbb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 0x55809fdbae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 0x55809fdbb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43014/0 (socket says 172.21.15.28:43014) 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7effd640 1 -- 172.21.15.28:0/2851743479 learned_addr learned my addr 172.21.15.28:0/2851743479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7ffff640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fdbb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc8c7be740 10 client.0 _ll_register_callbacks cb 0x7efc780095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 -- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 msgr2=0x55809fdbb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fdbb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 -- 172.21.15.28:0/2851743479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 msgr2=0x55809fdbb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 0x55809fdbb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.543+0000 7efc7f7fe640 1 -- 172.21.15.28:0/2851743479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55809fe23680 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc7f7fe640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 0x55809fdbae90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc7002a450 tx=0x7efc7002ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65c79e5691e64ad server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70035040 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc70004950 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc700054f0 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc70004d30 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 10 client.25622 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55809fe24860 con 0x55809f9b39f0 2024-09-21T05:47:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc700054f0 con 0x55809f9b39f0 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efc7005f020 con 0x55809f9b39f0 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 10 client.25622 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 10 client.25622 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 20 client.25622 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.544+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9f5070 0x55809fe26f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc8c7be740 4 client.25622 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55809fd108a0 con 0x55809f9f5070 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc7effd640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9f5070 0x55809fe26f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc5effd640 10 client.25622 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc7effd640 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9f5070 0x55809fe26f40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c3673ea450f4259 server_cookie=9b5a7fc2378be09f in_seq=0 out_seq=0 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc5effd640 1 -- 172.21.15.28:0/2851743479 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55809fd108a0 con 0x55809f9f5070 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc5effd640 10 client.25622 handle_command_reply: tid=0 2024-09-21T05:47:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc8c7be740 1 client.25622 shutdown 2024-09-21T05:47:48.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 msgr2=0x55809fdbae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.545+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 0x55809fdbae90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc7002a450 tx=0x7efc7002ab90 comp rx=0 tx=0).stop 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 shutdown_connections 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55809f9f5070 0x55809fe26f40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55809f9d4390 0x55809fdbb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55809f9cc1b0 0x55809fdbb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc5effd640 0 client.25622 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 --2- 172.21.15.28:0/2851743479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55809f9b39f0 0x55809fdbae90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 >> 172.21.15.28:0/2851743479 conn(0x55809f9e9460 msgr2=0x55809fda91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 shutdown_connections 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 1 -- 172.21.15.28:0/2851743479 wait complete. 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.546+0000 7efc8c7be740 20 client.25622 trim_cache size 0 max 16384 2024-09-21T05:47:48.552 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 msgr2=0x7efc680b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 0x7efc680b8b90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efc78005fd0 tx=0x7efc78009740 comp rx=0 tx=0).stop 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 msgr2=0x7efc8019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e9c0 tx=0x7efc7400ee80 comp rx=0 tx=0).stop 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 shutdown_connections 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc80108d10 0x7efc8019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc680b6680 0x7efc680b8b90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc801083c0 0x7efc8019aa50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 --2- 172.21.15.28:0/2272881063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc80102a80 0x7efc8019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 >> 172.21.15.28:0/2272881063 conn(0x7efc80079740 msgr2=0x7efc80101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 shutdown_connections 2024-09-21T05:47:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:48.547+0000 7efc8624a640 1 -- 172.21.15.28:0/2272881063 wait complete. 2024-09-21T05:47:49.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:48 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:47:49.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:48 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:47:49.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:48 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:48 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:47:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:48 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:47:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:48 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:49.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:48 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:47:49.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:48 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:47:49.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:48 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:47:49.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:47:49.632 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T05:47:49.704 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T05:47:49.747 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T05:47:49.749 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T05:47:49.749 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T05:47:49.793 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T05:47:49.797 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:47:49.797 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T05:47:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:49.814 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-21T05:47:49.815 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-21T05:47:49.815 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-21T05:47:49.877 INFO:tasks.workunit.client.0.smithi028.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-21T05:47:49.878 INFO:tasks.workunit.client.0.smithi028.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-21T05:47:49.878 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap -- retry EXIT 2024-09-21T05:47:49.878 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -rf ffsb 2024-09-21T05:47:49.879 INFO:tasks.workunit.client.0.smithi028.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-21T05:47:49.882 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into 'ffsb'... 2024-09-21T05:47:50.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 -- 172.21.15.28:0/219171487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480109df0 msgr2=0x7f0480110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 --2- 172.21.15.28:0/219171487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480109df0 0x7f0480110620 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ca0 tx=0x7f047802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 -- 172.21.15.28:0/219171487 shutdown_connections 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 --2- 172.21.15.28:0/219171487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480109df0 0x7f0480110620 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 --2- 172.21.15.28:0/219171487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 0x7f04801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 --2- 172.21.15.28:0/219171487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0480103b60 0x7f0480103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 -- 172.21.15.28:0/219171487 >> 172.21.15.28:0/219171487 conn(0x7f048007b850 msgr2=0x7f048007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 -- 172.21.15.28:0/219171487 shutdown_connections 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.273+0000 7f04869aa640 1 -- 172.21.15.28:0/219171487 wait complete. 2024-09-21T05:47:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 Processor -- start 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 -- start start 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 0x7f048019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0480109df0 0x7f048019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0480113ab0 con 0x7f0480103b60 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0480113930 con 0x7f0480109df0 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f04869aa640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0480113c30 con 0x7f04801094a0 2024-09-21T05:47:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f047ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f047ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43026/0 (socket says 172.21.15.28:43026) 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f047f7fe640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 0x7f048019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.274+0000 7f047ffff640 1 -- 172.21.15.28:0/3305812475 learned_addr learned my addr 172.21.15.28:0/3305812475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 -- 172.21.15.28:0/3305812475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 msgr2=0x7f048019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 0x7f048019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 -- 172.21.15.28:0/3305812475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0480109df0 msgr2=0x7f048019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0480109df0 0x7f048019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 -- 172.21.15.28:0/3305812475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04801a0370 con 0x7f0480103b60 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047ffff640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f047000d950 tx=0x7f047000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047d7fa640 1 -- 172.21.15.28:0/3305812475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470017070 con 0x7f0480103b60 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047d7fa640 1 -- 172.21.15.28:0/3305812475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0470009c30 con 0x7f0480103b60 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04801a0660 con 0x7f0480103b60 2024-09-21T05:47:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f047d7fa640 1 -- 172.21.15.28:0/3305812475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047000f620 con 0x7f0480103b60 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.275+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04801a6dc0 con 0x7f0480103b60 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 Processor -- start 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 -- start start 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f047d7fa640 1 -- 172.21.15.28:0/3305812475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f0470010460 con 0x7f0480103b60 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 0x560cda1d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1d6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cda14cea0 con 0x560cd9e0e390 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560cda13c2d0 con 0x560cd9e2f070 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.278+0000 7f048cf1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560cda1400e0 con 0x560cd9e061b0 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1d6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43032/0 (socket says 172.21.15.28:43032) 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 -- 172.21.15.28:0/2730205198 learned_addr learned my addr 172.21.15.28:0/2730205198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f0484f47640 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 0x560cda1d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 -- 172.21.15.28:0/2730205198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 msgr2=0x560cda1d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1d6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 -- 172.21.15.28:0/2730205198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 msgr2=0x560cda1d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 0x560cda1d66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 -- 172.21.15.28:0/2730205198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd9e9a650 con 0x560cd9e0e390 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f0484f47640 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 0x560cda1d66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047d7fa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 0x7f04680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0474009c20 tx=0x7f0474011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70fcd52e0a6507e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f0484f47640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 0x7f04680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f047d7fa640 1 -- 172.21.15.28:0/3305812475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f04700e7370 con 0x7f0480103b60 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f04667fc640 1 -- 172.21.15.28:0/2730205198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474004230 con 0x560cd9e0e390 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f04667fc640 1 -- 172.21.15.28:0/2730205198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04740043d0 con 0x560cd9e0e390 2024-09-21T05:47:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.279+0000 7f04667fc640 1 -- 172.21.15.28:0/2730205198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474004d50 con 0x560cd9e0e390 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 -- 172.21.15.28:0/2730205198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 msgr2=0x560cda1d7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f0484f47640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 0x7f04680b8b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f047802ade0 tx=0x7f0478009410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0474009c20 tx=0x7f0474011810 comp rx=0 tx=0).stop 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 -- 172.21.15.28:0/2730205198 shutdown_connections 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e0e390 0x560cda1d7120 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1d6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2730205198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e2f070 0x560cda1d66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.280+0000 7f048cf1e740 1 -- 172.21.15.28:0/2730205198 >> 172.21.15.28:0/2730205198 conn(0x560cd9e23460 msgr2=0x560cda1d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.281+0000 7f048cf1e740 1 -- 172.21.15.28:0/2730205198 shutdown_connections 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.281+0000 7f048cf1e740 1 -- 172.21.15.28:0/2730205198 wait complete. 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.281+0000 7f048cf1e740 1 -- start start 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.281+0000 7f048cf1e740 1 Processor -- start 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.281+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e0e390 0x560cda1e6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e2f070 0x560cda1e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cd9f9d700 con 0x560cd9e2f070 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560cda0d7160 con 0x560cd9e0e390 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560cda1e91b0 con 0x560cd9e061b0 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 10 client.0 _ll_register_callbacks cb 0x560cda0f09e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f0484f47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e2f070 0x560cda1e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51748/0 (socket says 172.21.15.28:51748) 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 -- 172.21.15.28:0/2204453213 learned_addr learned my addr 172.21.15.28:0/2204453213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e0e390 0x560cda1e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 -- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e0e390 msgr2=0x560cda1e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e0e390 0x560cda1e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 -- 172.21.15.28:0/2204453213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e2f070 msgr2=0x560cda1e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e2f070 0x560cda1e6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 -- 172.21.15.28:0/2204453213 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cda104c90 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f047ffff640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f04700e4b40 tx=0x7f04700eaa10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd2c8ddfbbb1e32 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f045f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470020030 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0470010750 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.282+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04740043d0 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f04700acfb0 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 10 client.25657 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cda1f4eb0 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04740043d0 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0470010950 con 0x560cd9e061b0 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 10 client.25657 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 10 client.25657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 20 client.25657 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9ded9f0 0x560cda1f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 4 client.25657 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.283+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cda14edf0 con 0x560cd9ded9f0 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.284+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9ded9f0 0x560cda1f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.284+0000 7f045f7fe640 10 client.25657 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.284+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9ded9f0 0x560cda1f83a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2db9a3293652ec0 server_cookie=1e81e5d24a27874b in_seq=0 out_seq=0 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f045f7fe640 1 -- 172.21.15.28:0/2204453213 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560cda14edf0 con 0x560cd9ded9f0 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f045f7fe640 10 client.25657 handle_command_reply: tid=0 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 client.25657 shutdown 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 msgr2=0x560cda1e6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f04700e4b40 tx=0x7f04700eaa10 comp rx=0 tx=0).stop 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 shutdown_connections 2024-09-21T05:47:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9ded9f0 0x560cda1f83a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e2f070 0x560cda1e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e0e390 0x560cda1e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f045f7fe640 0 client.25657 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2204453213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1e6040 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 >> 172.21.15.28:0/2204453213 conn(0x560cd9e23460 msgr2=0x560cda1d2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 shutdown_connections 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.285+0000 7f048cf1e740 1 -- 172.21.15.28:0/2204453213 wait complete. 2024-09-21T05:47:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.286+0000 7f048cf1e740 20 client.25657 trim_cache size 0 max 16384 2024-09-21T05:47:50.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.299+0000 7f048cf1e740 1 Processor -- start 2024-09-21T05:47:50.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.299+0000 7f048cf1e740 1 -- start start 2024-09-21T05:47:50.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.299+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.299+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9ded9f0 0x560cda1d7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.299+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e061b0 0x560cda1e3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f048cf1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04801ad460 con 0x560cd9e23460 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f048cf1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04801aefa0 con 0x560cd9e061b0 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f048cf1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f04801adb30 con 0x560cd9ded9f0 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43058/0 (socket says 172.21.15.28:43058) 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 -- 172.21.15.28:0/2160208021 learned_addr learned my addr 172.21.15.28:0/2160208021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047f7fe640 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9ded9f0 0x560cda1d7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f0484f47640 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e061b0 0x560cda1e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 -- 172.21.15.28:0/2160208021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9ded9f0 msgr2=0x560cda1d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9ded9f0 0x560cda1d7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 -- 172.21.15.28:0/2160208021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e061b0 msgr2=0x560cda1e3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e061b0 0x560cda1e3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 -- 172.21.15.28:0/2160208021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cda1e7300 con 0x560cd9e23460 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f047ffff640 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f04700ad170 tx=0x7f047000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a23c50a5c2e833 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.300+0000 7f0465ffb640 1 -- 172.21.15.28:0/2160208021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470014030 con 0x560cd9e23460 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f0465ffb640 1 -- 172.21.15.28:0/2160208021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04700abeb0 con 0x560cd9e23460 2024-09-21T05:47:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f0465ffb640 1 -- 172.21.15.28:0/2160208021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470020050 con 0x560cd9e23460 2024-09-21T05:47:50.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 -- 172.21.15.28:0/2160208021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 msgr2=0x560cda1d44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f04700ad170 tx=0x7f047000bf60 comp rx=0 tx=0).stop 2024-09-21T05:47:50.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 -- 172.21.15.28:0/2160208021 shutdown_connections 2024-09-21T05:47:50.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9e061b0 0x560cda1e3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9ded9f0 0x560cda1d7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 --2- 172.21.15.28:0/2160208021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda1d44b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.301+0000 7f048cf1e740 1 -- 172.21.15.28:0/2160208021 >> 172.21.15.28:0/2160208021 conn(0x560cda1f86f0 msgr2=0x560cda1d76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.302+0000 7f048cf1e740 1 -- 172.21.15.28:0/2160208021 shutdown_connections 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.302+0000 7f048cf1e740 1 -- 172.21.15.28:0/2160208021 wait complete. 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.302+0000 7f048cf1e740 1 -- start start 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.302+0000 7f048cf1e740 1 Processor -- start 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.302+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9ded9f0 0x560cda1ea820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda261db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04801ae8d0 con 0x560cd9e23460 2024-09-21T05:47:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04801ae200 con 0x560cd9ded9f0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f04801ab440 con 0x560cd9e061b0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f048cf1e740 10 client.0 _ll_register_callbacks cb 0x7f047000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51778/0 (socket says 172.21.15.28:51778) 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 -- 172.21.15.28:0/343483315 learned_addr learned my addr 172.21.15.28:0/343483315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f0484f47640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda261db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047ffff640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9ded9f0 0x560cda1ea820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 -- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9ded9f0 msgr2=0x560cda1ea820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9ded9f0 0x560cda1ea820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 -- 172.21.15.28:0/343483315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 msgr2=0x560cda261db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda261db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 -- 172.21.15.28:0/343483315 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cda2622f0 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f0484f47640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda261db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f047f7fe640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0474009b40 tx=0x7f0474009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb04384b580a490f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.303+0000 7f045f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474031020 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0474004250 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04700abeb0 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f047403c050 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 10 client.25663 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cda263410 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04700abeb0 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0474024040 con 0x560cd9e061b0 2024-09-21T05:47:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 10 client.25663 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 10 client.25663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 20 client.25663 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9e0e390 0x560cda265a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 4 client.25663 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.304+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560cda14a8a0 con 0x560cd9e0e390 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f047ffff640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9e0e390 0x560cda265a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f045f7fe640 10 client.25663 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f047ffff640 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9e0e390 0x560cda265a00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d53de3aecf69c8cf server_cookie=f2e465ad9610f7ef in_seq=0 out_seq=0 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f045f7fe640 1 -- 172.21.15.28:0/343483315 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560cda14a8a0 con 0x560cd9e0e390 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f045f7fe640 10 client.25663 handle_command_reply: tid=0 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.305+0000 7f048cf1e740 1 client.25663 shutdown 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 msgr2=0x560cda1ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0474009b40 tx=0x7f0474009e90 comp rx=0 tx=0).stop 2024-09-21T05:47:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 shutdown_connections 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x560cd9e0e390 0x560cda265a00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd9e23460 0x560cda261db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560cd9e061b0 0x560cda1ead60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f045f7fe640 0 client.25663 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 --2- 172.21.15.28:0/343483315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560cd9ded9f0 0x560cda1ea820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 >> 172.21.15.28:0/343483315 conn(0x560cda1f86f0 msgr2=0x560cda1e74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 shutdown_connections 2024-09-21T05:47:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 1 -- 172.21.15.28:0/343483315 wait complete. 2024-09-21T05:47:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.306+0000 7f048cf1e740 20 client.25663 trim_cache size 0 max 16384 2024-09-21T05:47:50.313 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 msgr2=0x7f04680b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 0x7f04680b8b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f047802ade0 tx=0x7f0478009410 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 msgr2=0x7f048019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f047000d950 tx=0x7f047000de10 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 shutdown_connections 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0480109df0 0x7f048019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04801094a0 0x7f048019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f04680b6680 0x7f04680b8b90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 --2- 172.21.15.28:0/3305812475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0480103b60 0x7f048019a400 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 >> 172.21.15.28:0/3305812475 conn(0x7f048007b850 msgr2=0x7f0480107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 shutdown_connections 2024-09-21T05:47:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:50.307+0000 7f04869aa640 1 -- 172.21.15.28:0/3305812475 wait complete. 2024-09-21T05:47:50.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:50 smithi181 ceph-mon[34819]: pgmap v349: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T05:47:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:50 smithi028 ceph-mon[31227]: pgmap v349: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T05:47:50.839 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap - EXIT 2024-09-21T05:47:50.839 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd ffsb 2024-09-21T05:47:50.839 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./configure 2024-09-21T05:47:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:50 smithi129 ceph-mon[34885]: pgmap v349: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T05:47:51.007 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-21T05:47:51.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:47:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.904+0000 7f19c44b5640 1 -- 172.21.15.28:0/2930139391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc1094a0 msgr2=0x7f19bc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.904+0000 7f19c44b5640 1 --2- 172.21.15.28:0/2930139391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc1094a0 0x7f19bc1098b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009900 tx=0x7f19ac02a410 comp rx=0 tx=0).stop 2024-09-21T05:47:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 -- 172.21.15.28:0/2930139391 shutdown_connections 2024-09-21T05:47:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 --2- 172.21.15.28:0/2930139391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc109df0 0x7f19bc110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 --2- 172.21.15.28:0/2930139391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc1094a0 0x7f19bc1098b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 --2- 172.21.15.28:0/2930139391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 -- 172.21.15.28:0/2930139391 >> 172.21.15.28:0/2930139391 conn(0x7f19bc07b850 msgr2=0x7f19bc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 -- 172.21.15.28:0/2930139391 shutdown_connections 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.905+0000 7f19c44b5640 1 -- 172.21.15.28:0/2930139391 wait complete. 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 Processor -- start 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 -- start start 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 0x7f19bc19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113ab0 con 0x7f19bc109df0 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113930 con 0x7f19bc103b60 2024-09-21T05:47:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c44b5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc113c30 con 0x7f19bc1094a0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43092/0 (socket says 172.21.15.28:43092) 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c2a52640 1 -- 172.21.15.28:0/646939984 learned_addr learned my addr 172.21.15.28:0/646939984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c2251640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.906+0000 7f19c1a50640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 0x7f19bc19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 -- 172.21.15.28:0/646939984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 msgr2=0x7f19bc19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 0x7f19bc19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 -- 172.21.15.28:0/646939984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 msgr2=0x7f19bc19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 -- 172.21.15.28:0/646939984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc1a03f0 con 0x7f19bc109df0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c1a50640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 0x7f19bc19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2251640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c2a52640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f19b400eaa0 tx=0x7f19b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19b37fe640 1 -- 172.21.15.28:0/646939984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b400cdb0 con 0x7f19bc109df0 2024-09-21T05:47:51.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19b37fe640 1 -- 172.21.15.28:0/646939984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b4005020 con 0x7f19bc109df0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc1a06e0 con 0x7f19bc109df0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19b37fe640 1 -- 172.21.15.28:0/646939984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b40116d0 con 0x7f19bc109df0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.907+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc1a6e40 con 0x7f19bc109df0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.910+0000 7f19caa29740 1 Processor -- start 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.910+0000 7f19b37fe640 1 -- 172.21.15.28:0/646939984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f19b40051d0 con 0x7f19bc109df0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.910+0000 7f19caa29740 1 -- start start 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 0x55a6e4a0c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 0x55a6e4a0cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a0d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6e4982ea0 con 0x55a6e463c1b0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a6e49722d0 con 0x55a6e4644390 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19caa29740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a6e49760e0 con 0x55a6e4665070 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c1a50640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 0x55a6e4a0c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c1a50640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 0x55a6e4a0c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51814/0 (socket says 172.21.15.28:51814) 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c1a50640 1 -- 172.21.15.28:0/2219985512 learned_addr learned my addr 172.21.15.28:0/2219985512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 0x55a6e4a0cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c1a50640 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a0d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 -- 172.21.15.28:0/2219985512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 msgr2=0x55a6e4a0c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 0x55a6e4a0c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 -- 172.21.15.28:0/2219985512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 msgr2=0x55a6e4a0d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a0d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.911+0000 7f19c2251640 1 -- 172.21.15.28:0/2219985512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6e46d0650 con 0x55a6e463c1b0 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19b37fe640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 0x7f19a80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19c2251640 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 0x55a6e4a0cbe0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f19b800bc80 tx=0x7f19b800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facaf64e5abb3ed server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19c1a50640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 0x7f19a80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19b37fe640 1 -- 172.21.15.28:0/646939984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f19b40e7900 con 0x7f19bc109df0 2024-09-21T05:47:51.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19b0d9a640 1 -- 172.21.15.28:0/2219985512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8015070 con 0x55a6e463c1b0 2024-09-21T05:47:51.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19b0d9a640 1 -- 172.21.15.28:0/2219985512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b801a910 con 0x55a6e463c1b0 2024-09-21T05:47:51.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.912+0000 7f19b0d9a640 1 -- 172.21.15.28:0/2219985512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8011860 con 0x55a6e463c1b0 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 -- 172.21.15.28:0/2219985512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 msgr2=0x55a6e4a0cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 0x55a6e4a0cbe0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f19b800bc80 tx=0x7f19b800f730 comp rx=0 tx=0).stop 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19c1a50640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 0x7f19a80b8c10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19ac032fd0 tx=0x7f19ac031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 -- 172.21.15.28:0/2219985512 shutdown_connections 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a0d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e463c1b0 0x55a6e4a0cbe0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 --2- 172.21.15.28:0/2219985512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4665070 0x55a6e4a0c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 -- 172.21.15.28:0/2219985512 >> 172.21.15.28:0/2219985512 conn(0x55a6e4659460 msgr2=0x55a6e4a0a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 -- 172.21.15.28:0/2219985512 shutdown_connections 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.913+0000 7f19caa29740 1 -- 172.21.15.28:0/2219985512 wait complete. 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 -- start start 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 Processor -- start 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a1c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6e47d3700 con 0x55a6e4665070 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a6e490d160 con 0x55a6e4644390 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a6e4a1f1e0 con 0x55a6e463c1b0 2024-09-21T05:47:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.914+0000 7f19caa29740 10 client.0 _ll_register_callbacks cb 0x55a6e49269e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43116/0 (socket says 172.21.15.28:43116) 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 -- 172.21.15.28:0/1040651884 learned_addr learned my addr 172.21.15.28:0/1040651884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c1a50640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a1c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2251640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 -- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 msgr2=0x55a6e4a1c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 -- 172.21.15.28:0/1040651884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 msgr2=0x55a6e4a1c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a1c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 -- 172.21.15.28:0/1040651884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6e493ac90 con 0x55a6e4665070 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19c2a52640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f19b40e5670 tx=0x7f19b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9951a8011ac8843 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b401f040 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f19b40ac440 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.915+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b801a910 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19b40b2250 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 10 client.16029 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6e4a2aeb0 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b801a910 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f19b400f040 con 0x55a6e4665070 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 10 client.16029 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 20 client.16029 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e46239f0 0x55a6e4a2e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 4 client.16029 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.916+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6e4984df0 con 0x55a6e46239f0 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.917+0000 7f19c2251640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e46239f0 0x55a6e4a2e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.918+0000 7f19c2251640 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e46239f0 0x55a6e4a2e3d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f02a353b070634 server_cookie=2ae0241617425fa3 in_seq=0 out_seq=0 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.918+0000 7f19a5ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19a5ffb640 1 -- 172.21.15.28:0/1040651884 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6e4984df0 con 0x55a6e46239f0 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19a5ffb640 10 client.16029 handle_command_reply: tid=0 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 client.16029 shutdown 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 msgr2=0x55a6e4a1cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f19b40e5670 tx=0x7f19b40057f0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 shutdown_connections 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e46239f0 0x55a6e4a2e3d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a1cb00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4644390 0x55a6e4a1c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 --2- 172.21.15.28:0/1040651884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 >> 172.21.15.28:0/1040651884 conn(0x55a6e4659460 msgr2=0x55a6e4a0e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19a5ffb640 0 client.16029 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 shutdown_connections 2024-09-21T05:47:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 1 -- 172.21.15.28:0/1040651884 wait complete. 2024-09-21T05:47:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.919+0000 7f19caa29740 20 client.16029 trim_cache size 0 max 16384 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 Processor -- start 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 -- start start 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4644390 0x55a6e4a19270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1ad4f0 con 0x55a6e4665070 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1af030 con 0x55a6e463c1b0 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19caa29740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1adbc0 con 0x55a6e4644390 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43128/0 (socket says 172.21.15.28:43128) 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 -- 172.21.15.28:0/649056411 learned_addr learned my addr 172.21.15.28:0/649056411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c1a50640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4644390 0x55a6e4a19270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2251640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 -- 172.21.15.28:0/649056411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4644390 msgr2=0x55a6e4a19270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4644390 0x55a6e4a19270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 -- 172.21.15.28:0/649056411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 msgr2=0x55a6e4a1dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2a52640 1 -- 172.21.15.28:0/649056411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6e4a127e0 con 0x55a6e4665070 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.933+0000 7f19c2251640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.934+0000 7f19c2a52640 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f19b40e5120 tx=0x7f19b40b65e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53633f3be43b81 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.934+0000 7f19a7fff640 1 -- 172.21.15.28:0/649056411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b401f070 con 0x55a6e4665070 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.934+0000 7f19a7fff640 1 -- 172.21.15.28:0/649056411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40b7190 con 0x55a6e4665070 2024-09-21T05:47:51.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.934+0000 7f19a7fff640 1 -- 172.21.15.28:0/649056411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b40aece0 con 0x55a6e4665070 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- 172.21.15.28:0/649056411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 msgr2=0x55a6e4a197b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f19b40e5120 tx=0x7f19b40b65e0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- 172.21.15.28:0/649056411 shutdown_connections 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4665070 0x55a6e4a197b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e4644390 0x55a6e4a19270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 --2- 172.21.15.28:0/649056411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e463c1b0 0x55a6e4a1dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- 172.21.15.28:0/649056411 >> 172.21.15.28:0/649056411 conn(0x55a6e4659460 msgr2=0x55a6e4a1d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- 172.21.15.28:0/649056411 shutdown_connections 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- 172.21.15.28:0/649056411 wait complete. 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 -- start start 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.935+0000 7f19caa29740 1 Processor -- start 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a24df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 0x55a6e4a20940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1ae960 con 0x55a6e4644390 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1ae290 con 0x55a6e4665070 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1ab4d0 con 0x55a6e463c1b0 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 10 client.0 _ll_register_callbacks cb 0x7f19b40a9e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c2251640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a24df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43130/0 (socket says 172.21.15.28:43130) 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 -- 172.21.15.28:0/3867191962 learned_addr learned my addr 172.21.15.28:0/3867191962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c2251640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a24df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51844/0 (socket says 172.21.15.28:51844) 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c2a52640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 0x55a6e4a20940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 -- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 msgr2=0x55a6e4a24df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a24df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 -- 172.21.15.28:0/3867191962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 msgr2=0x55a6e4a20940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 0x55a6e4a20940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 -- 172.21.15.28:0/3867191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6e4a210d0 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c2a52640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 0x55a6e4a20940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19c1a50640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f19bc06c9f0 tx=0x7f19ac009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c058385f3e5c3e3f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac035040 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f19ac004450 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.936+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b40b7190 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19ac0035f0 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 10 client.16035 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6e4a98db0 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40b7190 con 0x55a6e4644390 2024-09-21T05:47:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f19ac047070 con 0x55a6e4644390 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 10 client.16035 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 20 client.16035 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e45f8da0 0x55a6e4a9b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 4 client.16035 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a6e49808a0 con 0x55a6e45f8da0 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.937+0000 7f19c2251640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e45f8da0 0x55a6e4a9b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19c2251640 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e45f8da0 0x55a6e4a9b5f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25db663636b9d7bd server_cookie=3d14ace83c75b6a9 in_seq=0 out_seq=0 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19a5ffb640 10 client.16035 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19a5ffb640 1 -- 172.21.15.28:0/3867191962 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a6e49808a0 con 0x55a6e45f8da0 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19a5ffb640 10 client.16035 handle_command_reply: tid=0 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 client.16035 shutdown 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 msgr2=0x55a6e4a202a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f19bc06c9f0 tx=0x7f19ac009260 comp rx=0 tx=0).stop 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 shutdown_connections 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55a6e45f8da0 0x55a6e4a9b5f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a6e4665070 0x55a6e4a20940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6e4644390 0x55a6e4a202a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 --2- 172.21.15.28:0/3867191962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6e463c1b0 0x55a6e4a24df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 >> 172.21.15.28:0/3867191962 conn(0x55a6e4659460 msgr2=0x55a6e4a0d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19a5ffb640 0 client.16035 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 shutdown_connections 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.938+0000 7f19caa29740 1 -- 172.21.15.28:0/3867191962 wait complete. 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19caa29740 20 client.16035 trim_cache size 0 max 16384 2024-09-21T05:47:51.945 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 msgr2=0x7f19a80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 0x7f19a80b8c10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19ac032fd0 tx=0x7f19ac031400 comp rx=0 tx=0).stop 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 msgr2=0x7f19bc19fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f19b400eaa0 tx=0x7f19b400ef60 comp rx=0 tx=0).stop 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 shutdown_connections 2024-09-21T05:47:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f19a80b6700 0x7f19a80b8c10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19bc109df0 0x7f19bc19fcc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19bc1094a0 0x7f19bc19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 --2- 172.21.15.28:0/646939984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc103b60 0x7f19bc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 >> 172.21.15.28:0/646939984 conn(0x7f19bc07b850 msgr2=0x7f19bc107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 shutdown_connections 2024-09-21T05:47:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:51.939+0000 7f19c44b5640 1 -- 172.21.15.28:0/646939984 wait complete. 2024-09-21T05:47:52.015 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-09-21T05:47:52.023 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-21T05:47:52.024 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-09-21T05:47:52.038 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-09-21T05:47:52.068 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-21T05:47:52.068 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-21T05:47:52.079 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-09-21T05:47:52.084 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-09-21T05:47:52.152 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-09-21T05:47:52.154 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-09-21T05:47:52.157 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-09-21T05:47:52.193 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-09-21T05:47:52.236 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-09-21T05:47:52.267 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-09-21T05:47:52.318 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-09-21T05:47:52.497 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-21T05:47:52.606 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-09-21T05:47:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:52 smithi028 ceph-mon[31227]: pgmap v350: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:52.732 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-09-21T05:47:52.775 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-21T05:47:52.776 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-09-21T05:47:52.785 INFO:tasks.workunit.client.0.smithi028.stdout:checking for AIX... no 2024-09-21T05:47:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:52 smithi181 ceph-mon[34819]: pgmap v350: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:52.803 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-09-21T05:47:52.811 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-21T05:47:52.812 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-21T05:47:52.812 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-21T05:47:52.812 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-21T05:47:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:52 smithi129 ceph-mon[34885]: pgmap v350: 97 pgs: 97 active+clean; 1.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-21T05:47:52.869 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-09-21T05:47:52.930 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-09-21T05:47:53.099 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-09-21T05:47:53.176 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-21T05:47:53.221 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-09-21T05:47:53.267 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-09-21T05:47:53.323 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-09-21T05:47:53.372 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-09-21T05:47:53.419 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-09-21T05:47:53.467 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-09-21T05:47:53.514 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-09-21T05:47:53.562 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-09-21T05:47:53.610 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-09-21T05:47:53.678 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-09-21T05:47:53.700 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-09-21T05:47:53.700 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-09-21T05:47:53.733 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:47:53.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:47:53.752 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-09-21T05:47:53.773 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-09-21T05:47:53.773 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-09-21T05:47:53.825 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-09-21T05:47:53.845 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-09-21T05:47:53.845 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-09-21T05:47:53.860 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-09-21T05:47:53.910 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-09-21T05:47:53.930 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-09-21T05:47:53.931 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-09-21T05:47:53.945 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-09-21T05:47:53.997 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:54.001 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-09-21T05:47:54.027 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-09-21T05:47:54.027 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-09-21T05:47:54.085 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-09-21T05:47:54.109 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-09-21T05:47:54.110 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-09-21T05:47:54.153 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-09-21T05:47:54.193 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-21T05:47:54.194 INFO:tasks.workunit.client.0.smithi028.stdout:checking for function prototypes... yes 2024-09-21T05:47:54.252 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether setvbuf arguments are reversed... no 2024-09-21T05:47:54.332 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-09-21T05:47:54.401 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-09-21T05:47:54.471 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-09-21T05:47:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 -- 172.21.15.28:0/347907249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc60108d20 msgr2=0x7efc6010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 --2- 172.21.15.28:0/347907249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc60108d20 0x7efc6010f550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efc50009ca0 tx=0x7efc5002a8e0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 -- 172.21.15.28:0/347907249 shutdown_connections 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 --2- 172.21.15.28:0/347907249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc60108d20 0x7efc6010f550 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 --2- 172.21.15.28:0/347907249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc601083d0 0x7efc601087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 --2- 172.21.15.28:0/347907249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 0x7efc60102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 -- 172.21.15.28:0/347907249 >> 172.21.15.28:0/347907249 conn(0x7efc600797b0 msgr2=0x7efc60079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 -- 172.21.15.28:0/347907249 shutdown_connections 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.522+0000 7efc6767b640 1 -- 172.21.15.28:0/347907249 wait complete. 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.523+0000 7efc6767b640 1 Processor -- start 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.523+0000 7efc6767b640 1 -- start start 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.523+0000 7efc6767b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 0x7efc6019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc6767b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc6767b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc60108d20 0x7efc6019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc6767b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc60113b40 con 0x7efc601083d0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc6767b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc601139c0 con 0x7efc60108d20 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc6767b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc60113cc0 con 0x7efc60102ae0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43144/0 (socket says 172.21.15.28:43144) 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 -- 172.21.15.28:0/2849916884 learned_addr learned my addr 172.21.15.28:0/2849916884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc65417640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 0x7efc6019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc65c18640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc60108d20 0x7efc6019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 -- 172.21.15.28:0/2849916884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 msgr2=0x7efc6019a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 0x7efc6019a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 -- 172.21.15.28:0/2849916884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc60108d20 msgr2=0x7efc6019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc60108d20 0x7efc6019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 -- 172.21.15.28:0/2849916884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc601a0440 con 0x7efc601083d0 2024-09-21T05:47:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc64c16640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efc54006fd0 tx=0x7efc5400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.524+0000 7efc5e7fc640 1 -- 172.21.15.28:0/2849916884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54011830 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.525+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc601a0730 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.525+0000 7efc5e7fc640 1 -- 172.21.15.28:0/2849916884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc54016070 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.525+0000 7efc5e7fc640 1 -- 172.21.15.28:0/2849916884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54010e00 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.525+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc601a6f10 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.527+0000 7efc6dbef740 1 Processor -- start 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.527+0000 7efc6dbef740 1 -- start start 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc5e7fc640 1 -- 172.21.15.28:0/2849916884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7efc5401b460 con 0x7efc601083d0 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b87430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b87eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c653afde60 con 0x55c6537bf670 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c653aed290 con 0x55c6537e0350 2024-09-21T05:47:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc6dbef740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c653af10a0 con 0x55c6537b7490 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b87eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51874/0 (socket says 172.21.15.28:51874) 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 -- 172.21.15.28:0/165219035 learned_addr learned my addr 172.21.15.28:0/165219035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65c18640 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b87430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 -- 172.21.15.28:0/165219035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 msgr2=0x55c653b87430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b87430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 -- 172.21.15.28:0/165219035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 msgr2=0x55c653b87eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b87eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65417640 1 -- 172.21.15.28:0/165219035 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65384b930 con 0x55c6537b7490 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.528+0000 7efc65c18640 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b87430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc65417640 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc58009d20 tx=0x7efc58011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76a83e32e477d5b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc5e7fc640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 0x7efc480b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc477fe640 1 -- 172.21.15.28:0/165219035 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5801e070 con 0x55c6537b7490 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc65c18640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 0x7efc480b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc477fe640 1 -- 172.21.15.28:0/165219035 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc58019ae0 con 0x55c6537b7490 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc477fe640 1 -- 172.21.15.28:0/165219035 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5800ea60 con 0x55c6537b7490 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.529+0000 7efc5e7fc640 1 -- 172.21.15.28:0/2849916884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc540e6bb0 con 0x7efc601083d0 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- 172.21.15.28:0/165219035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 msgr2=0x55c653b87970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc58009d20 tx=0x7efc58011810 comp rx=0 tx=0).stop 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc65c18640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 0x7efc480b8b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efc50006010 tx=0x7efc50009840 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- 172.21.15.28:0/165219035 shutdown_connections 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b87eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b87970 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 --2- 172.21.15.28:0/165219035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b87430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- 172.21.15.28:0/165219035 >> 172.21.15.28:0/165219035 conn(0x55c6537d4740 msgr2=0x55c653b85320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- 172.21.15.28:0/165219035 shutdown_connections 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- 172.21.15.28:0/165219035 wait complete. 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.530+0000 7efc6dbef740 1 -- start start 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 Processor -- start 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b93000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 0x55c653b9ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c65394e890 con 0x55c6537bf670 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c653a87d40 con 0x55c6537b7490 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c653b95c10 con 0x55c6537e0350 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc6dbef740 10 client.0 _ll_register_callbacks cb 0x55c653aebc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43158/0 (socket says 172.21.15.28:43158) 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc64c16640 1 -- 172.21.15.28:0/3426209146 learned_addr learned my addr 172.21.15.28:0/3426209146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.531+0000 7efc65c18640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 0x55c653b9ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc65417640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b93000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 -- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 msgr2=0x55c653b9ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 0x55c653b9ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 -- 172.21.15.28:0/3426209146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 msgr2=0x55c653b93000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b93000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 -- 172.21.15.28:0/3426209146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c653b9f520 con 0x55c6537bf670 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc65c18640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 0x55c653b9ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc64c16640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efc54009fd0 tx=0x7efc540ea2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b88a13000208a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc540a88c0 con 0x55c6537bf670 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc540a8570 con 0x55c6537bf670 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc58019ae0 con 0x55c6537bf670 2024-09-21T05:47:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc540ac190 con 0x55c6537bf670 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc6dbef740 10 client.16047 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c653ba1eb0 con 0x55c6537bf670 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.532+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc58019ae0 con 0x55c6537bf670 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efc540b1080 con 0x55c6537bf670 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 10 client.16047 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 20 client.16047 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c65379ecd0 0x55c653ba5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 4 client.16047 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c653affdb0 con 0x55c65379ecd0 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc65417640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c65379ecd0 0x55c653ba5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc44ff9640 10 client.16047 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.533+0000 7efc65417640 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c65379ecd0 0x55c653ba5360 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e35ccd35b5d96d51 server_cookie=83c51ba2f420b8d3 in_seq=0 out_seq=0 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc44ff9640 1 -- 172.21.15.28:0/3426209146 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c653affdb0 con 0x55c65379ecd0 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc44ff9640 10 client.16047 handle_command_reply: tid=0 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 client.16047 shutdown 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 msgr2=0x55c653b93560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efc54009fd0 tx=0x7efc540ea2f0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 shutdown_connections 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c65379ecd0 0x55c653ba5360 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537e0350 0x55c653b9ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b93560 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3426209146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b93000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 >> 172.21.15.28:0/3426209146 conn(0x55c6537d4740 msgr2=0x55c653b8a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc44ff9640 0 client.16047 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 shutdown_connections 2024-09-21T05:47:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 1 -- 172.21.15.28:0/3426209146 wait complete. 2024-09-21T05:47:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.536+0000 7efc6dbef740 20 client.16047 trim_cache size 0 max 16384 2024-09-21T05:47:54.551 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 Processor -- start 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 -- start start 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b9ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b8fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc601af750 con 0x55c6537bf670 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc601b1290 con 0x55c6537e0350 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc6dbef740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc601afe20 con 0x55c6537b7490 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc64c16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43174/0 (socket says 172.21.15.28:43174) 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc64c16640 1 -- 172.21.15.28:0/3015246735 learned_addr learned my addr 172.21.15.28:0/3015246735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc65417640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b9ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.555+0000 7efc65c18640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b8fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 -- 172.21.15.28:0/3015246735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 msgr2=0x55c653b8fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b8fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 -- 172.21.15.28:0/3015246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 msgr2=0x55c653b8d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 -- 172.21.15.28:0/3015246735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c653b89720 con 0x55c6537b7490 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc64c16640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc65417640 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b9ee30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efc5801c040 tx=0x7efc58009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=116e441776b808d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc46ffd640 1 -- 172.21.15.28:0/3015246735 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5801e040 con 0x55c6537b7490 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc46ffd640 1 -- 172.21.15.28:0/3015246735 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc5801f040 con 0x55c6537b7490 2024-09-21T05:47:54.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.556+0000 7efc46ffd640 1 -- 172.21.15.28:0/3015246735 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5800ea60 con 0x55c6537b7490 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 -- 172.21.15.28:0/3015246735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 msgr2=0x55c653b9ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b9ee30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efc5801c040 tx=0x7efc58009620 comp rx=0 tx=0).stop 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 -- 172.21.15.28:0/3015246735 shutdown_connections 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537e0350 0x55c653b8fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537bf670 0x55c653b8d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 --2- 172.21.15.28:0/3015246735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537b7490 0x55c653b9ee30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 -- 172.21.15.28:0/3015246735 >> 172.21.15.28:0/3015246735 conn(0x55c6537d4740 msgr2=0x55c653b91680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 -- 172.21.15.28:0/3015246735 shutdown_connections 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.557+0000 7efc6dbef740 1 -- 172.21.15.28:0/3015246735 wait complete. 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 -- start start 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 Processor -- start 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 0x55c653c0ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc601b0bc0 con 0x55c6537e0350 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc601b04f0 con 0x55c6537b7490 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc601ad730 con 0x55c6537bf670 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc6dbef740 10 client.0 _ll_register_callbacks cb 0x7efc540ea3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.558+0000 7efc65c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 0x55c653c0ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc65c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 0x55c653c0ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43176/0 (socket says 172.21.15.28:43176) 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51898/0 (socket says 172.21.15.28:51898) 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc65c18640 1 -- 172.21.15.28:0/303103900 learned_addr learned my addr 172.21.15.28:0/303103900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc65417640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 -- 172.21.15.28:0/303103900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 msgr2=0x55c653b976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 -- 172.21.15.28:0/303103900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 msgr2=0x55c653c0ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 0x55c653c0ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 -- 172.21.15.28:0/303103900 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c653c0f390 con 0x55c6537bf670 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc64c16640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efc5400bde0 tx=0x7efc54010820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c641c276f58b8b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc540ac370 con 0x55c6537bf670 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7efc540b2690 con 0x55c6537bf670 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.559+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc5801f040 con 0x55c6537bf670 2024-09-21T05:47:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc540ac7f0 con 0x55c6537bf670 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 10 client.25699 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c653c10650 con 0x55c6537bf670 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc5801f040 con 0x55c6537bf670 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efc540b2890 con 0x55c6537bf670 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 10 client.25699 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 10 client.25699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 20 client.25699 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c653774080 0x55c653c12d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 4 client.25699 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c653afb860 con 0x55c653774080 2024-09-21T05:47:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.560+0000 7efc65417640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c653774080 0x55c653c12d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.561+0000 7efc44ff9640 10 client.25699 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.561+0000 7efc65417640 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c653774080 0x55c653c12d80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8082b0970626a003 server_cookie=383eb89c5a9a2fbb in_seq=0 out_seq=0 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc44ff9640 1 -- 172.21.15.28:0/303103900 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c653afb860 con 0x55c653774080 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc44ff9640 10 client.25699 handle_command_reply: tid=0 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 client.25699 shutdown 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 msgr2=0x55c653b97cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efc5400bde0 tx=0x7efc54010820 comp rx=0 tx=0).stop 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 shutdown_connections 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55c653774080 0x55c653c12d80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc44ff9640 0 client.25699 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6537e0350 0x55c653c0ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c6537bf670 0x55c653b97cc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 --2- 172.21.15.28:0/303103900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c6537b7490 0x55c653b976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.562+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 >> 172.21.15.28:0/303103900 conn(0x55c6537d4740 msgr2=0x55c653b91240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 shutdown_connections 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6dbef740 1 -- 172.21.15.28:0/303103900 wait complete. 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6dbef740 20 client.25699 trim_cache size 0 max 16384 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "778a6c00-a466-4250-b8ce-492ba85cbd6a", 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:47:54.569 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 msgr2=0x7efc480b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 0x7efc480b8b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efc50006010 tx=0x7efc50009840 comp rx=0 tx=0).stop 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 msgr2=0x7efc6019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efc54006fd0 tx=0x7efc5400b8e0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 shutdown_connections 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc60108d20 0x7efc6019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7efc480b6680 0x7efc480b8b90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc601083d0 0x7efc6019aa70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 --2- 172.21.15.28:0/2849916884 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc60102ae0 0x7efc6019a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.563+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 >> 172.21.15.28:0/2849916884 conn(0x7efc600797b0 msgr2=0x7efc601043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.564+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 shutdown_connections 2024-09-21T05:47:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:54.564+0000 7efc6767b640 1 -- 172.21.15.28:0/2849916884 wait complete. 2024-09-21T05:47:54.650 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-09-21T05:47:54.744 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-09-21T05:47:54.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:54 smithi181 ceph-mon[34819]: pgmap v351: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 144 KiB/s wr, 19 op/s 2024-09-21T05:47:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:54 smithi028 ceph-mon[31227]: pgmap v351: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 144 KiB/s wr, 19 op/s 2024-09-21T05:47:54.856 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-09-21T05:47:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:54 smithi129 ceph-mon[34885]: pgmap v351: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 144 KiB/s wr, 19 op/s 2024-09-21T05:47:54.919 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-09-21T05:47:54.982 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-09-21T05:47:55.006 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-09-21T05:47:55.107 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-09-21T05:47:55.121 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-21T05:47:55.139 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-09-21T05:47:55.187 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-09-21T05:47:55.248 INFO:tasks.workunit.client.0.smithi028.stderr:+ make 2024-09-21T05:47:55.290 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-21T05:47:55.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-21T05:47:55.293 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-21T05:47:55.293 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-21T05:47:55.293 INFO:tasks.workunit.client.0.smithi028.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-21T05:47:55.294 INFO:tasks.workunit.client.0.smithi028.stderr: any GNU archive site. 2024-09-21T05:47:55.295 INFO:tasks.workunit.client.0.smithi028.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-21T05:47:55.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-21T05:47:55.297 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-21T05:47:55.297 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-21T05:47:55.298 INFO:tasks.workunit.client.0.smithi028.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-21T05:47:55.298 INFO:tasks.workunit.client.0.smithi028.stderr: Grab them from any GNU archive site. 2024-09-21T05:47:55.350 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-21T05:47:55.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:47:55.646 INFO:tasks.workunit.client.0.smithi028.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-21T05:47:55.646 INFO:tasks.workunit.client.0.smithi028.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-21T05:47:55.646 INFO:tasks.workunit.client.0.smithi028.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-21T05:47:55.646 INFO:tasks.workunit.client.0.smithi028.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-21T05:47:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:55.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:55 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:47:55.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:55 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:47:55.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.775 INFO:tasks.workunit.client.0.smithi028.stdout:/bin/sh ./config.status --recheck 2024-09-21T05:47:55.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:55 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:47:55.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:55 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:47:55.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:55 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:47:55.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:55 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:47:55.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:47:55.828 INFO:tasks.workunit.client.0.smithi028.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-09-21T05:47:55.931 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 -- 172.21.15.28:0/1364822358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 msgr2=0x7fae40110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 --2- 172.21.15.28:0/1364822358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40110300 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fae38009ca0 tx=0x7fae3802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 -- 172.21.15.28:0/1364822358 shutdown_connections 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 --2- 172.21.15.28:0/1364822358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40110300 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 --2- 172.21.15.28:0/1364822358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae40105300 0x7fae4010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 --2- 172.21.15.28:0/1364822358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae401049b0 0x7fae40104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 -- 172.21.15.28:0/1364822358 >> 172.21.15.28:0/1364822358 conn(0x7fae40079730 msgr2=0x7fae40079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 -- 172.21.15.28:0/1364822358 shutdown_connections 2024-09-21T05:47:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.221+0000 7fae469c4640 1 -- 172.21.15.28:0/1364822358 wait complete. 2024-09-21T05:47:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 Processor -- start 2024-09-21T05:47:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 -- start start 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 0x7fae401026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae40113ae0 con 0x7fae4010de10 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae40113960 con 0x7fae40105300 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.223+0000 7fae469c4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae40113c60 con 0x7fae401049b0 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae44f61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53422/0 (socket says 172.21.15.28:53422) 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 -- 172.21.15.28:0/2219250811 learned_addr learned my addr 172.21.15.28:0/2219250811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3ffff640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 0x7fae401026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 -- 172.21.15.28:0/2219250811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 msgr2=0x7fae401026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 0x7fae401026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 -- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 msgr2=0x7fae40103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3f7fe640 1 -- 172.21.15.28:0/2219250811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae401a8730 con 0x7fae40105300 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae44f61640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.224+0000 7fae3ffff640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 0x7fae401026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae3400e9c0 tx=0x7fae3400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae3d7fa640 1 -- 172.21.15.28:0/2219250811 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3400ccc0 con 0x7fae40105300 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae401a8a20 con 0x7fae40105300 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae3d7fa640 1 -- 172.21.15.28:0/2219250811 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae340054d0 con 0x7fae40105300 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae401a8f60 con 0x7fae40105300 2024-09-21T05:47:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.225+0000 7fae3d7fa640 1 -- 172.21.15.28:0/2219250811 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34010430 con 0x7fae40105300 2024-09-21T05:47:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.229+0000 7fae4cf38740 1 Processor -- start 2024-09-21T05:47:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.229+0000 7fae4cf38740 1 -- start start 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c92cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c92d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f71c8a2e50 con 0x55f71c55c1b0 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f71c892280 con 0x55f71c564390 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae4cf38740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f71c896090 con 0x55f71c585070 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c92cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c92cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35672/0 (socket says 172.21.15.28:35672) 2024-09-21T05:47:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae3ffff640 1 -- 172.21.15.28:0/3075269128 learned_addr learned my addr 172.21.15.28:0/3075269128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.230+0000 7fae44f61640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae3d7fa640 1 -- 172.21.15.28:0/2219250811 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fae34005020 con 0x7fae40105300 2024-09-21T05:47:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c92d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae44f61640 1 -- 172.21.15.28:0/3075269128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 msgr2=0x55f71c92d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae44f61640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c92d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae44f61640 1 -- 172.21.15.28:0/3075269128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 msgr2=0x55f71c92cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae44f61640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c92cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae44f61640 1 -- 172.21.15.28:0/3075269128 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f71c5f0650 con 0x55f71c585070 2024-09-21T05:47:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.231+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c92d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:47:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.232+0000 7fae44f61640 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92c650 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae38005e30 tx=0x7fae38009950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca971374dea3df1e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.233+0000 7fae267fc640 1 -- 172.21.15.28:0/3075269128 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38031650 con 0x55f71c585070 2024-09-21T05:47:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.233+0000 7fae267fc640 1 -- 172.21.15.28:0/3075269128 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae38036040 con 0x55f71c585070 2024-09-21T05:47:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.233+0000 7fae267fc640 1 -- 172.21.15.28:0/3075269128 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38004510 con 0x55f71c585070 2024-09-21T05:47:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.233+0000 7fae3d7fa640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 0x7fae280b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.233+0000 7fae3ffff640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 0x7fae280b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.234+0000 7fae3d7fa640 1 -- 172.21.15.28:0/2219250811 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fae340e7af0 con 0x7fae40105300 2024-09-21T05:47:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.234+0000 7fae4cf38740 1 -- 172.21.15.28:0/3075269128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 msgr2=0x55f71c92c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.234+0000 7fae3ffff640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 0x7fae280b8c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55f71c92b2d0 tx=0x7fae3000b640 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.234+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92c650 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae38005e30 tx=0x7fae38009950 comp rx=0 tx=0).stop 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 -- 172.21.15.28:0/3075269128 shutdown_connections 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c92d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c92cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3075269128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92c650 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 -- 172.21.15.28:0/3075269128 >> 172.21.15.28:0/3075269128 conn(0x55f71c579460 msgr2=0x55f71c92a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 -- 172.21.15.28:0/3075269128 shutdown_connections 2024-09-21T05:47:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 -- 172.21.15.28:0/3075269128 wait complete. 2024-09-21T05:47:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.235+0000 7fae4cf38740 1 -- start start 2024-09-21T05:47:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.236+0000 7fae4cf38740 1 Processor -- start 2024-09-21T05:47:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.236+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 0x55f71c937d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.236+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.236+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c943f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae4cf38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f71c6f3700 con 0x55f71c585070 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 0x55f71c937d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae4cf38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f71c82d110 con 0x55f71c55c1b0 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae4cf38740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f71c93ac50 con 0x55f71c564390 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:34824/0 (socket says 172.21.15.28:34824) 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1643355310 learned_addr learned my addr 172.21.15.28:0/1643355310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 0x55f71c937d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53446/0 (socket says 172.21.15.28:53446) 2024-09-21T05:47:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae4cf38740 10 client.0 _ll_register_callbacks cb 0x55f71c846990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:56.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1643355310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 msgr2=0x55f71c937d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 0x55f71c937d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1643355310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 msgr2=0x55f71c943f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:47:56.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c943f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.237+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1643355310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71c85ac40 con 0x55f71c564390 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c6a0 tx=0x7fae340aa080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0f3e72cd5cf4aa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae340ad070 con 0x55f71c564390 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fae340a9250 con 0x55f71c564390 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae38036040 con 0x55f71c564390 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae340a9070 con 0x55f71c564390 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae4cf38740 10 client.25711 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:56.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.238+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f71c9469e0 con 0x55f71c564390 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae38036040 con 0x55f71c564390 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fae340a9450 con 0x55f71c564390 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 10 client.25711 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 10 client.25711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 20 client.25711 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c5439f0 0x55f71c949eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 4 client.25711 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.239+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f71c8a4da0 con 0x55f71c5439f0 2024-09-21T05:47:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.240+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c5439f0 0x55f71c949eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.240+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c5439f0 0x55f71c949eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2b6365d668a12b server_cookie=844a8f9365080cbb in_seq=0 out_seq=0 2024-09-21T05:47:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.240+0000 7fae1f7fe640 10 client.25711 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.241+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1643355310 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f71c8a4da0 con 0x55f71c5439f0 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.241+0000 7fae1f7fe640 10 client.25711 handle_command_reply: tid=0 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.241+0000 7fae4cf38740 1 client.25711 shutdown 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.241+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 msgr2=0x55f71c9382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.241+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c6a0 tx=0x7fae340aa080 comp rx=0 tx=0).stop 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 shutdown_connections 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c5439f0 0x55f71c949eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c943f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c564390 0x55f71c9382d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae1f7fe640 0 client.25711 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1643355310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c55c1b0 0x55f71c937d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 >> 172.21.15.28:0/1643355310 conn(0x55f71c579460 msgr2=0x55f71c92f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 shutdown_connections 2024-09-21T05:47:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 1 -- 172.21.15.28:0/1643355310 wait complete. 2024-09-21T05:47:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.242+0000 7fae4cf38740 20 client.25711 trim_cache size 0 max 16384 2024-09-21T05:47:56.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.265+0000 7fae4cf38740 1 Processor -- start 2024-09-21T05:47:56.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.265+0000 7fae4cf38740 1 -- start start 2024-09-21T05:47:56.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.265+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c9392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c929b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae4cf38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae401b1720 con 0x55f71c55c1b0 2024-09-21T05:47:56.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae4cf38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae401b3260 con 0x55f71c564390 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae4cf38740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae401b1df0 con 0x55f71c585070 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c929b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c929b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53450/0 (socket says 172.21.15.28:53450) 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae3f7fe640 1 -- 172.21.15.28:0/3635734514 learned_addr learned my addr 172.21.15.28:0/3635734514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae3ffff640 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c9392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 -- 172.21.15.28:0/3635734514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 msgr2=0x55f71c929b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c929b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 -- 172.21.15.28:0/3635734514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 msgr2=0x55f71c9392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c9392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.266+0000 7fae44f61640 1 -- 172.21.15.28:0/3635734514 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f71c93a160 con 0x55f71c585070 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.267+0000 7fae44f61640 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92a0b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae3802aeb0 tx=0x7fae38009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33ea03bfc194a53 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.267+0000 7fae25ffb640 1 -- 172.21.15.28:0/3635734514 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38055070 con 0x55f71c585070 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.267+0000 7fae25ffb640 1 -- 172.21.15.28:0/3635734514 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae38036050 con 0x55f71c585070 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.267+0000 7fae25ffb640 1 -- 172.21.15.28:0/3635734514 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae380042a0 con 0x55f71c585070 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 -- 172.21.15.28:0/3635734514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 msgr2=0x55f71c92a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92a0b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae3802aeb0 tx=0x7fae38009540 comp rx=0 tx=0).stop 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 -- 172.21.15.28:0/3635734514 shutdown_connections 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c585070 0x55f71c92a0b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c929b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 --2- 172.21.15.28:0/3635734514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c55c1b0 0x55f71c9392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.268+0000 7fae4cf38740 1 -- 172.21.15.28:0/3635734514 >> 172.21.15.28:0/3635734514 conn(0x55f71c579460 msgr2=0x55f71c92e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.269+0000 7fae4cf38740 1 -- 172.21.15.28:0/3635734514 shutdown_connections 2024-09-21T05:47:56.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.269+0000 7fae4cf38740 1 -- 172.21.15.28:0/3635734514 wait complete. 2024-09-21T05:47:56.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.269+0000 7fae4cf38740 1 -- start start 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 Processor -- start 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c55c1b0 0x55f71c93bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c93c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae401b2b90 con 0x55f71c585070 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae401b24c0 con 0x55f71c564390 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae401af700 con 0x55f71c55c1b0 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae4cf38740 10 client.0 _ll_register_callbacks cb 0x7fae3400ced0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53460/0 (socket says 172.21.15.28:53460) 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.270+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1191668752 learned_addr learned my addr 172.21.15.28:0/1191668752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae44f61640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c93c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c55c1b0 0x55f71c93bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c55c1b0 msgr2=0x55f71c93bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c55c1b0 0x55f71c93bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1191668752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 msgr2=0x55f71c93c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c93c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 -- 172.21.15.28:0/1191668752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71c9b3990 con 0x55f71c564390 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae3f7fe640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fae340e7e20 tx=0x7fae340a9df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12a692fd215e48ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.271+0000 7fae1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae340ad070 con 0x55f71c564390 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fae340c3ab0 con 0x55f71c564390 2024-09-21T05:47:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae38036050 con 0x55f71c564390 2024-09-21T05:47:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae340ad6d0 con 0x55f71c564390 2024-09-21T05:47:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae4cf38740 10 client.25676 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.272+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f71c9b4c20 con 0x55f71c564390 2024-09-21T05:47:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae38036050 con 0x55f71c564390 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fae340c3cb0 con 0x55f71c564390 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 10 client.25676 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 10 client.25676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 20 client.25676 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c518da0 0x55f71c9b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 4 client.25676 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f71c8a0850 con 0x55f71c518da0 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.273+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c518da0 0x55f71c9b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.274+0000 7fae3ffff640 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c518da0 0x55f71c9b71c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ad498d4ebd138e server_cookie=f438157a4b37b18e in_seq=0 out_seq=0 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.274+0000 7fae1f7fe640 10 client.25676 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.274+0000 7fae1f7fe640 1 -- 172.21.15.28:0/1191668752 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1226 (crc 0 0 0) 0x55f71c8a0850 con 0x55f71c518da0 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.274+0000 7fae1f7fe640 10 client.25676 handle_command_reply: tid=0 2024-09-21T05:47:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.274+0000 7fae4cf38740 1 client.25676 shutdown 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 msgr2=0x55f71c93c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fae340e7e20 tx=0x7fae340a9df0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 shutdown_connections 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f71c518da0 0x55f71c9b71c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f71c585070 0x55f71c93c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae1f7fe640 0 client.25676 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f71c564390 0x55f71c93c490 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 --2- 172.21.15.28:0/1191668752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f71c55c1b0 0x55f71c93bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 >> 172.21.15.28:0/1191668752 conn(0x55f71c579460 msgr2=0x55f71c92e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 shutdown_connections 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 1 -- 172.21.15.28:0/1191668752 wait complete. 2024-09-21T05:47:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.275+0000 7fae4cf38740 20 client.25676 trim_cache size 0 max 16384 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:56.281 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:47:56.283 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:47:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 msgr2=0x7fae280b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 0x7fae280b8c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55f71c92b2d0 tx=0x7fae3000b640 comp rx=0 tx=0).stop 2024-09-21T05:47:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 msgr2=0x7fae40102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae3400e9c0 tx=0x7fae3400ee80 comp rx=0 tx=0).stop 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 shutdown_connections 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fae280b6700 0x7fae280b8c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4010de10 0x7fae40103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae40105300 0x7fae40102c30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 --2- 172.21.15.28:0/2219250811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae401049b0 0x7fae401026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.276+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 >> 172.21.15.28:0/2219250811 conn(0x7fae40079730 msgr2=0x7fae400ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.277+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 shutdown_connections 2024-09-21T05:47:56.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:56.277+0000 7fae469c4640 1 -- 172.21.15.28:0/2219250811 wait complete. 2024-09-21T05:47:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:56 smithi181 ceph-mon[34819]: pgmap v352: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 146 KiB/s wr, 17 op/s 2024-09-21T05:47:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:56 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:47:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:56 smithi028 ceph-mon[31227]: pgmap v352: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 146 KiB/s wr, 17 op/s 2024-09-21T05:47:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:56 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:47:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:56 smithi129 ceph-mon[34885]: pgmap v352: 97 pgs: 97 active+clean; 3.1 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 146 KiB/s wr, 17 op/s 2024-09-21T05:47:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:56 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:47:56.927 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:47:56.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:47:56.938 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-09-21T05:47:56.943 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-21T05:47:56.943 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-09-21T05:47:56.971 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-09-21T05:47:57.000 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-21T05:47:57.000 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-21T05:47:57.012 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-09-21T05:47:57.019 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-09-21T05:47:57.106 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-09-21T05:47:57.106 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-09-21T05:47:57.146 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-09-21T05:47:57.187 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-09-21T05:47:57.189 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:57.226 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-09-21T05:47:57.252 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-09-21T05:47:57.275 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-09-21T05:47:57.353 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-21T05:47:57.463 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-09-21T05:47:57.536 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-09-21T05:47:57.584 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-21T05:47:57.586 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- 172.21.15.28:0/920775163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0104880 msgr2=0x7f1ae0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/920775163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0104880 0x7f1ae0104c90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009900 tx=0x7f1ad002a410 comp rx=0 tx=0).stop 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- 172.21.15.28:0/920775163 shutdown_connections 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/920775163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae010de30 0x7f1ae0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/920775163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae01052b0 0x7f1ae010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/920775163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0104880 0x7f1ae0104c90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- 172.21.15.28:0/920775163 >> 172.21.15.28:0/920775163 conn(0x7f1ae0079730 msgr2=0x7f1ae0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- 172.21.15.28:0/920775163 shutdown_connections 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- 172.21.15.28:0/920775163 wait complete. 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 Processor -- start 2024-09-21T05:47:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.662+0000 7f1ae66dd640 1 -- start start 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae0104880 0x7f1ae0102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae01052b0 0x7f1ae0102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae0113a70 con 0x7f1ae010de30 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01138f0 con 0x7f1ae01052b0 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae66dd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae0113bf0 con 0x7f1ae0104880 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35726/0 (socket says 172.21.15.28:35726) 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 -- 172.21.15.28:0/422170800 learned_addr learned my addr 172.21.15.28:0/422170800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae01052b0 0x7f1ae0102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 -- 172.21.15.28:0/422170800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae0104880 msgr2=0x7f1ae0102790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae0104880 0x7f1ae0102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 -- 172.21.15.28:0/422170800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae01052b0 msgr2=0x7f1ae0102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae01052b0 0x7f1ae0102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.663+0000 7f1ae4c7a640 1 -- 172.21.15.28:0/422170800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae01a87c0 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1ae4c7a640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800efd0 tx=0x7f1ad800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8003b70 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad8003d10 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae01a8ab0 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80108a0 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.664+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae01a8fa0 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.665+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae0104c90 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.667+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1ad8004b40 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.668+0000 7f1add7fa640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 0x7f1ac80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.668+0000 7f1adffff640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 0x7f1ac80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.668+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1ad80e8030 con 0x7f1ae010de30 2024-09-21T05:47:57.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.669+0000 7f1adffff640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 0x7f1ac80b8b90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1ad002a930 tx=0x7f1ad0031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:57.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.676+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1ad80ae540 con 0x7f1ae010de30 2024-09-21T05:47:57.722 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-09-21T05:47:57.759 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-09-21T05:47:57.797 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-09-21T05:47:57.836 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-09-21T05:47:57.875 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-09-21T05:47:57.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.906+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ae0103a90 con 0x7f1ae010de30 2024-09-21T05:47:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.907+0000 7f1add7fa640 1 -- 172.21.15.28:0/422170800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11986 (secure 0 0 0) 0x7f1ae0103a90 con 0x7f1ae010de30 2024-09-21T05:47:57.910 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:47:57.910 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"btime":"2024-09-21T05:46:30:445939+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:46:30.084799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:47:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-21T05:47:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 msgr2=0x7f1ac80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 0x7f1ac80b8b90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1ad002a930 tx=0x7f1ad0031400 comp rx=0 tx=0).stop 2024-09-21T05:47:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 msgr2=0x7f1ae0103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800efd0 tx=0x7f1ad800c6a0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 shutdown_connections 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ac80b6680 0x7f1ac80b8b90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae010de30 0x7f1ae0103210 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ae01052b0 0x7f1ae0102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 --2- 172.21.15.28:0/422170800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae0104880 0x7f1ae0102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 >> 172.21.15.28:0/422170800 conn(0x7f1ae0079730 msgr2=0x7f1ae0100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 shutdown_connections 2024-09-21T05:47:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:57.910+0000 7f1ae66dd640 1 -- 172.21.15.28:0/422170800 wait complete. 2024-09-21T05:47:57.916 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-09-21T05:47:57.959 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-09-21T05:47:58.001 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-09-21T05:47:58.051 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-09-21T05:47:58.093 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-09-21T05:47:58.153 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h usability... no 2024-09-21T05:47:58.182 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h presence... no 2024-09-21T05:47:58.182 INFO:tasks.workunit.client.0.smithi028.stdout:checking for minix/config.h... no 2024-09-21T05:47:58.219 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-21T05:47:58.220 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-09-21T05:47:58.257 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-21T05:47:58.257 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-21T05:47:58.258 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-21T05:47:58.258 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-21T05:47:58.302 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-09-21T05:47:58.346 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-09-21T05:47:58.347 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... (cached) yes 2024-09-21T05:47:58.424 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-21T05:47:58.509 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-09-21T05:47:58.528 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-09-21T05:47:58.528 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-09-21T05:47:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:58 smithi028 ceph-mon[31227]: pgmap v353: 97 pgs: 97 active+clean; 3.7 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 225 KiB/s wr, 40 op/s 2024-09-21T05:47:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:47:58 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/422170800' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:58.575 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-09-21T05:47:58.594 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-09-21T05:47:58.594 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-09-21T05:47:58.669 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-09-21T05:47:58.690 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-09-21T05:47:58.690 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-09-21T05:47:58.704 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-09-21T05:47:58.745 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-09-21T05:47:58.767 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-09-21T05:47:58.767 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-09-21T05:47:58.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:47:58.782 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-09-21T05:47:58.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:58 smithi181 ceph-mon[34819]: pgmap v353: 97 pgs: 97 active+clean; 3.7 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 225 KiB/s wr, 40 op/s 2024-09-21T05:47:58.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:47:58 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/422170800' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:58.819 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-09-21T05:47:58.836 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-09-21T05:47:58.836 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-09-21T05:47:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:58 smithi129 ceph-mon[34885]: pgmap v353: 97 pgs: 97 active+clean; 3.7 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 225 KiB/s wr, 40 op/s 2024-09-21T05:47:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:47:58 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/422170800' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:47:58.887 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-09-21T05:47:58.917 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-09-21T05:47:58.917 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-09-21T05:47:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 -- 172.21.15.28:0/4200552307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc10dd60 msgr2=0x7f5bdc110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 --2- 172.21.15.28:0/4200552307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc110250 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009ca0 tx=0x7f5bd402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 -- 172.21.15.28:0/4200552307 shutdown_connections 2024-09-21T05:47:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 --2- 172.21.15.28:0/4200552307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc110250 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 --2- 172.21.15.28:0/4200552307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc105250 0x7f5bdc10d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 --2- 172.21.15.28:0/4200552307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc104900 0x7f5bdc104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 -- 172.21.15.28:0/4200552307 >> 172.21.15.28:0/4200552307 conn(0x7f5bdc07a740 msgr2=0x7f5bdc07ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 -- 172.21.15.28:0/4200552307 shutdown_connections 2024-09-21T05:47:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.940+0000 7f5be121d640 1 -- 172.21.15.28:0/4200552307 wait complete. 2024-09-21T05:47:58.944 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 Processor -- start 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 -- start start 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 0x7f5bdc102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1139e0 con 0x7f5bdc105250 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc113860 con 0x7f5bdc10dd60 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.941+0000 7f5be121d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc113b60 con 0x7f5bdc104900 2024-09-21T05:47:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 0x7f5bdc102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:34864/0 (socket says 172.21.15.28:34864) 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bdad9d640 1 -- 172.21.15.28:0/34717484 learned_addr learned my addr 172.21.15.28:0/34717484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 -- 172.21.15.28:0/34717484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 msgr2=0x7f5bdc102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 -- 172.21.15.28:0/34717484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc10dd60 msgr2=0x7f5bdc1030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc1030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 -- 172.21.15.28:0/34717484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc1a86d0 con 0x7f5bdc105250 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bda59c640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 0x7f5bdc102b70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc06c9c0 tx=0x7f5bd000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.942+0000 7f5bc3fff640 1 -- 172.21.15.28:0/34717484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd000cc40 con 0x7f5bdc105250 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.943+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc1a89c0 con 0x7f5bdc105250 2024-09-21T05:47:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.943+0000 7f5bc3fff640 1 -- 172.21.15.28:0/34717484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd000cde0 con 0x7f5bdc105250 2024-09-21T05:47:58.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.943+0000 7f5bc3fff640 1 -- 172.21.15.28:0/34717484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0004a00 con 0x7f5bdc105250 2024-09-21T05:47:58.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.943+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc1a8ec0 con 0x7f5bdc105250 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.945+0000 7f5be8263740 1 Processor -- start 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.945+0000 7f5be8263740 1 -- start start 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c825131b0 0x562c828b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c8282d6f0 con 0x562c825131b0 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c8281cb20 con 0x562c824f24d0 2024-09-21T05:47:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5be8263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c82820930 con 0x562c824ea2f0 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bc3fff640 1 -- 172.21.15.28:0/34717484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5bd0004ba0 con 0x7f5bdc105250 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:34880/0 (socket says 172.21.15.28:34880) 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 -- 172.21.15.28:0/1336552381 learned_addr learned my addr 172.21.15.28:0/1336552381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c825131b0 0x562c828b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 -- 172.21.15.28:0/1336552381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 msgr2=0x562c828b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 -- 172.21.15.28:0/1336552381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c825131b0 msgr2=0x562c828b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c825131b0 0x562c828b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 -- 172.21.15.28:0/1336552381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c827e2530 con 0x562c824ea2f0 2024-09-21T05:47:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.946+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00f4a0 tx=0x7f5bcc00f960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df245d261c3a930 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bc17fa640 1 -- 172.21.15.28:0/1336552381 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc01d070 con 0x562c824ea2f0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bc17fa640 1 -- 172.21.15.28:0/1336552381 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc010db0 con 0x562c824ea2f0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bc17fa640 1 -- 172.21.15.28:0/1336552381 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00d420 con 0x562c824ea2f0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bc3fff640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 0x7f5bc40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 0x7f5bc40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.947+0000 7f5bc3fff640 1 -- 172.21.15.28:0/34717484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5bd00e7630 con 0x7f5bdc105250 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 0x7f5bc40b8bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd402f3b0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 -- 172.21.15.28:0/1336552381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 msgr2=0x562c828b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00f4a0 tx=0x7f5bcc00f960 comp rx=0 tx=0).stop 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 -- 172.21.15.28:0/1336552381 shutdown_connections 2024-09-21T05:47:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828b9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828b8ef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 --2- 172.21.15.28:0/1336552381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c825131b0 0x562c828b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 -- 172.21.15.28:0/1336552381 >> 172.21.15.28:0/1336552381 conn(0x562c82507300 msgr2=0x562c828b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 -- 172.21.15.28:0/1336552381 shutdown_connections 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.948+0000 7f5be8263740 1 -- 172.21.15.28:0/1336552381 wait complete. 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 -- start start 2024-09-21T05:47:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 Processor -- start 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824ea2f0 0x562c828c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828c4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c82754ab0 con 0x562c824ea2f0 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c827c2730 con 0x562c824f24d0 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c828c7130 con 0x562c825131b0 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.949+0000 7f5be8263740 10 client.0 _ll_register_callbacks cb 0x562c828166b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824ea2f0 0x562c828c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:34896/0 (socket says 172.21.15.28:34896) 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 -- 172.21.15.28:0/1927515391 learned_addr learned my addr 172.21.15.28:0/1927515391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bda59c640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 -- 172.21.15.28:0/1927515391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 msgr2=0x562c828c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 -- 172.21.15.28:0/1927515391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824ea2f0 msgr2=0x562c828c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824ea2f0 0x562c828c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 -- 172.21.15.28:0/1927515391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8257e790 con 0x562c825131b0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bda59c640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009ca0 tx=0x7f5bd402a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ce6dec4feee59b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4004040 con 0x562c825131b0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5bd40041e0 con 0x562c825131b0 2024-09-21T05:47:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.950+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc010db0 con 0x562c825131b0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5bd40043e0 con 0x562c825131b0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 10 client.25729 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c825f7620 con 0x562c825131b0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc010db0 con 0x562c825131b0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5bd405a070 con 0x562c825131b0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 10 client.25729 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 10 client.25729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 20 client.25729 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c824d1d80 0x562c828d6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 4 client.25729 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c8282f640 con 0x562c824d1d80 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.951+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c824d1d80 0x562c828d6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.952+0000 7f5bbeff5640 10 client.25729 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.952+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c824d1d80 0x562c828d6260 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4741f8bcdb5babad server_cookie=7b7612019b5a0177 in_seq=0 out_seq=0 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5bbeff5640 1 -- 172.21.15.28:0/1927515391 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562c8282f640 con 0x562c824d1d80 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5bbeff5640 10 client.25729 handle_command_reply: tid=0 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 client.25729 shutdown 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 msgr2=0x562c828d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009ca0 tx=0x7f5bd402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 shutdown_connections 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c824d1d80 0x562c828d6260 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c825131b0 0x562c828d0420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 --2- 172.21.15.28:0/1927515391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824ea2f0 0x562c828c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5bbeff5640 0 client.25729 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 >> 172.21.15.28:0/1927515391 conn(0x562c82507300 msgr2=0x562c828b99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 shutdown_connections 2024-09-21T05:47:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 1 -- 172.21.15.28:0/1927515391 wait complete. 2024-09-21T05:47:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.953+0000 7f5be8263740 20 client.25729 trim_cache size 0 max 16384 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 Processor -- start 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 -- start start 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b1830 con 0x562c824d1d80 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b3370 con 0x562c824f24d0 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5be8263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b1f00 con 0x562c824ea2f0 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5bda59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.968+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bda59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:34902/0 (socket says 172.21.15.28:34902) 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35774/0 (socket says 172.21.15.28:35774) 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bda59c640 1 -- 172.21.15.28:0/3786450036 learned_addr learned my addr 172.21.15.28:0/3786450036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 -- 172.21.15.28:0/3786450036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 msgr2=0x562c828c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 -- 172.21.15.28:0/3786450036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 msgr2=0x562c828c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 -- 172.21.15.28:0/3786450036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c828bacb0 con 0x562c824f24d0 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bda59c640 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828d0320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4043040 tx=0x7f5bd4009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=271636d5aa7bc59b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bc0ff9640 1 -- 172.21.15.28:0/3786450036 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4060020 con 0x562c824f24d0 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bc0ff9640 1 -- 172.21.15.28:0/3786450036 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4036050 con 0x562c824f24d0 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.969+0000 7f5bc0ff9640 1 -- 172.21.15.28:0/3786450036 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4030820 con 0x562c824f24d0 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 -- 172.21.15.28:0/3786450036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 msgr2=0x562c828d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828d0320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4043040 tx=0x7f5bd4009440 comp rx=0 tx=0).stop 2024-09-21T05:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 -- 172.21.15.28:0/3786450036 shutdown_connections 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824f24d0 0x562c828d0320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824ea2f0 0x562c828c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 --2- 172.21.15.28:0/3786450036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824d1d80 0x562c828c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 -- 172.21.15.28:0/3786450036 >> 172.21.15.28:0/3786450036 conn(0x562c82507300 msgr2=0x562c828c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 -- 172.21.15.28:0/3786450036 shutdown_connections 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.970+0000 7f5be8263740 1 -- 172.21.15.28:0/3786450036 wait complete. 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 -- start start 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 Processor -- start 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 0x562c828cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824ea2f0 0x562c828c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b2ca0 con 0x562c824f24d0 2024-09-21T05:47:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b25d0 con 0x562c824ea2f0 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1af810 con 0x562c824d1d80 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5be8263740 10 client.0 _ll_register_callbacks cb 0x7f5bd4009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5bdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 0x562c828cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5bdb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35778/0 (socket says 172.21.15.28:35778) 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5bdb59e640 1 -- 172.21.15.28:0/3046182110 learned_addr learned my addr 172.21.15.28:0/3046182110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.971+0000 7f5bda59c640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824ea2f0 0x562c828c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 -- 172.21.15.28:0/3046182110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824ea2f0 msgr2=0x562c828c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824ea2f0 0x562c828c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 -- 172.21.15.28:0/3046182110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 msgr2=0x562c828c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 -- 172.21.15.28:0/3046182110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c828c8aa0 con 0x562c824d1d80 2024-09-21T05:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdb59e640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bdad9d640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 0x562c828cca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006010 tx=0x7f5bcc018a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0fa29986bc4b94 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc01d040 con 0x562c824d1d80 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd4036050 con 0x562c824d1d80 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5bcc018680 con 0x562c824d1d80 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc038050 con 0x562c824d1d80 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5be8263740 10 client.25735 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.972+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c82940a60 con 0x562c824d1d80 2024-09-21T05:47:58.979 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-21T05:47:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4036050 con 0x562c824d1d80 2024-09-21T05:47:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5bcc010820 con 0x562c824d1d80 2024-09-21T05:47:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 10 client.25735 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 20 client.25735 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c825131b0 0x562c82942e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 4 client.25735 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c8282b0f0 con 0x562c825131b0 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.973+0000 7f5bda59c640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c825131b0 0x562c82942e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.974+0000 7f5bbeff5640 10 client.25735 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:58.974+0000 7f5bda59c640 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c825131b0 0x562c82942e90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df841bcfb66eefe3 server_cookie=a9b182d4e3458bd3 in_seq=0 out_seq=0 2024-09-21T05:47:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:47:59.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5bbeff5640 1 -- 172.21.15.28:0/3046182110 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c8282b0f0 con 0x562c825131b0 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5bbeff5640 10 client.25735 handle_command_reply: tid=0 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 client.25735 shutdown 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 msgr2=0x562c828cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 0x562c828cca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006010 tx=0x7f5bcc018a10 comp rx=0 tx=0).stop 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 shutdown_connections 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c825131b0 0x562c82942e90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c824f24d0 0x562c828c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5bbeff5640 0 client.25735 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c824ea2f0 0x562c828c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 --2- 172.21.15.28:0/3046182110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c824d1d80 0x562c828cca50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 >> 172.21.15.28:0/3046182110 conn(0x562c82507300 msgr2=0x562c828c5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 shutdown_connections 2024-09-21T05:47:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 1 -- 172.21.15.28:0/3046182110 wait complete. 2024-09-21T05:47:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.096+0000 7f5be8263740 20 client.25735 trim_cache size 0 max 16384 2024-09-21T05:47:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:47:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:47:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:47:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 msgr2=0x7f5bc40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 0x7f5bc40b8bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd402f3b0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 msgr2=0x7f5bdc102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 0x7f5bdc102b70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc06c9c0 tx=0x7f5bd000ee50 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 shutdown_connections 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bdc10dd60 0x7f5bdc1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5bc40b66b0 0x7f5bc40b8bc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bdc105250 0x7f5bdc102b70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 --2- 172.21.15.28:0/34717484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc104900 0x7f5bdc102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 >> 172.21.15.28:0/34717484 conn(0x7f5bdc07a740 msgr2=0x7f5bdc0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 shutdown_connections 2024-09-21T05:47:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:47:59.097+0000 7f5be121d640 1 -- 172.21.15.28:0/34717484 wait complete. 2024-09-21T05:47:59.142 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-09-21T05:47:59.199 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-09-21T05:47:59.255 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-09-21T05:47:59.311 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-09-21T05:47:59.367 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-09-21T05:47:59.426 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-09-21T05:47:59.482 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-09-21T05:47:59.537 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-09-21T05:47:59.596 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-09-21T05:47:59.620 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 -- 172.21.15.28:0/1797611894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc105270 msgr2=0x7ffabc10d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 --2- 172.21.15.28:0/1797611894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc105270 0x7ffabc10d8b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009900 tx=0x7ffaac02a410 comp rx=0 tx=0).stop 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 -- 172.21.15.28:0/1797611894 shutdown_connections 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 --2- 172.21.15.28:0/1797611894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 --2- 172.21.15.28:0/1797611894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc105270 0x7ffabc10d8b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 --2- 172.21.15.28:0/1797611894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc104840 0x7ffabc104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 -- 172.21.15.28:0/1797611894 >> 172.21.15.28:0/1797611894 conn(0x7ffabc079730 msgr2=0x7ffabc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 -- 172.21.15.28:0/1797611894 shutdown_connections 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 -- 172.21.15.28:0/1797611894 wait complete. 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.652+0000 7ffac3d03640 1 Processor -- start 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 -- start start 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc105270 0x7ffabc19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc113a70 con 0x7ffabc104840 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1138f0 con 0x7ffabc10ddf0 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac3d03640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc113bf0 con 0x7ffabc105270 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac1a9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac1a9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35804/0 (socket says 172.21.15.28:35804) 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac1a9f640 1 -- 172.21.15.28:0/2176290179 learned_addr learned my addr 172.21.15.28:0/2176290179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac129e640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc105270 0x7ffabc19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.653+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 -- 172.21.15.28:0/2176290179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc105270 msgr2=0x7ffabc19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc105270 0x7ffabc19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 -- 172.21.15.28:0/2176290179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 msgr2=0x7ffabc19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc19fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 -- 172.21.15.28:0/2176290179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffabc1a0490 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc19fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac1a9f640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ffab800ebc0 tx=0x7ffab800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2176290179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8003b70 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2176290179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab8003d10 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffabc1a0780 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2176290179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8019800 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.654+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffabc1a6ee0 con 0x7ffabc104840 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.657+0000 7ffaca277740 1 Processor -- start 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 -- start start 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9d71b0 0x5558abda7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abda7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558abd1dea0 con 0x5558aba00070 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5558abd0d2d0 con 0x5558ab9d71b0 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffaca277740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5558abd110e0 con 0x5558ab9df390 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35806/0 (socket says 172.21.15.28:35806) 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.658+0000 7ffac22a0640 1 -- 172.21.15.28:0/437457264 learned_addr learned my addr 172.21.15.28:0/437457264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2176290179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ffab80199a0 con 0x7ffabc104840 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac129e640 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abda7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 -- 172.21.15.28:0/437457264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 msgr2=0x5558abda7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abda7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 -- 172.21.15.28:0/437457264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9d71b0 msgr2=0x5558abda7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9d71b0 0x5558abda7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 -- 172.21.15.28:0/437457264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558aba6b650 con 0x5558aba00070 2024-09-21T05:47:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffac22a0640 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffab400bca0 tx=0x7ffab400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7abddaf6a219c69 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffaa7fff640 1 -- 172.21.15.28:0/437457264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4015070 con 0x5558aba00070 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffaa7fff640 1 -- 172.21.15.28:0/437457264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab40054d0 con 0x5558aba00070 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.659+0000 7ffaa7fff640 1 -- 172.21.15.28:0/437457264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab40116a0 con 0x5558aba00070 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.660+0000 7ffab2ffd640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 0x7ffaa80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.660+0000 7ffac1a9f640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 0x7ffaa80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.660+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2176290179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffab80e7fb0 con 0x7ffabc104840 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- 172.21.15.28:0/437457264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 msgr2=0x5558abda72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffac1a9f640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 0x7ffaa80b8c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5558abda5f40 tx=0x7ffab8077870 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffab400bca0 tx=0x7ffab400f730 comp rx=0 tx=0).stop 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- 172.21.15.28:0/437457264 shutdown_connections 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abda7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9d71b0 0x5558abda7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 --2- 172.21.15.28:0/437457264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abda72c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- 172.21.15.28:0/437457264 >> 172.21.15.28:0/437457264 conn(0x5558ab9f4460 msgr2=0x5558abda51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- 172.21.15.28:0/437457264 shutdown_connections 2024-09-21T05:47:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- 172.21.15.28:0/437457264 wait complete. 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 -- start start 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.661+0000 7ffaca277740 1 Processor -- start 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abdb2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abdbed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558abb6e700 con 0x5558ab9d71b0 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5558abca8160 con 0x5558aba00070 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5558abdb5aa0 con 0x5558ab9df390 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffaca277740 10 client.0 _ll_register_callbacks cb 0x5558abcc19e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35812/0 (socket says 172.21.15.28:35812) 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 -- 172.21.15.28:0/2385488863 learned_addr learned my addr 172.21.15.28:0/2385488863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac129e640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abdb2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abdbed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 -- 172.21.15.28:0/2385488863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 msgr2=0x5558abdb2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abdb2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 -- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 msgr2=0x5558abdbed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abdbed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.662+0000 7ffac22a0640 1 -- 172.21.15.28:0/2385488863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558abcd5c90 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffab4005c80 tx=0x7ffab4005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b6471a6115e6cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4015070 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffab40050c0 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab40054d0 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffab4027450 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaca277740 10 client.16104 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558abdc17b0 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab40054d0 con 0x5558ab9d71b0 2024-09-21T05:47:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.663+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffab402e020 con 0x5558ab9d71b0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 10 client.16104 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 10 client.16104 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 20 client.16104 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab9be9f0 0x5558abdc4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 4 client.16104 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558abd1fdf0 con 0x5558ab9be9f0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffac129e640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab9be9f0 0x5558abdc4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffac129e640 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab9be9f0 0x5558abdc4b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=219e3624ad70f9d5 server_cookie=7d42f61ef94ecab1 in_seq=0 out_seq=0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.664+0000 7ffaa57fa640 10 client.16104 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.665+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2385488863 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5558abd1fdf0 con 0x5558ab9be9f0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.665+0000 7ffaa57fa640 10 client.16104 handle_command_reply: tid=0 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.665+0000 7ffaca277740 1 client.16104 shutdown 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.665+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 msgr2=0x5558abdb29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.665+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffab4005c80 tx=0x7ffab4005760 comp rx=0 tx=0).stop 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 shutdown_connections 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab9be9f0 0x5558abdc4b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abdbed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abdb2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 --2- 172.21.15.28:0/2385488863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abdb29e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaa57fa640 0 client.16104 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 >> 172.21.15.28:0/2385488863 conn(0x5558ab9f4460 msgr2=0x5558abdaa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 shutdown_connections 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 1 -- 172.21.15.28:0/2385488863 wait complete. 2024-09-21T05:47:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.666+0000 7ffaca277740 20 client.16104 trim_cache size 0 max 16384 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 Processor -- start 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 -- start start 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9d71b0 0x5558abda8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9df390 0x5558abdb4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1af840 con 0x5558aba00070 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1b1380 con 0x5558ab9df390 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffaca277740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1aff10 con 0x5558ab9d71b0 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9d71b0 0x5558abda8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35826/0 (socket says 172.21.15.28:35826) 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.680+0000 7ffac22a0640 1 -- 172.21.15.28:0/3453411262 learned_addr learned my addr 172.21.15.28:0/3453411262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac129e640 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9df390 0x5558abdb4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 -- 172.21.15.28:0/3453411262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9d71b0 msgr2=0x5558abda8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9d71b0 0x5558abda8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 -- 172.21.15.28:0/3453411262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9df390 msgr2=0x5558abdb4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9df390 0x5558abdb4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 -- 172.21.15.28:0/3453411262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558abd9f460 con 0x5558aba00070 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffac22a0640 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ffab40118c0 tx=0x7ffab4005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982362461589f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffaa77fe640 1 -- 172.21.15.28:0/3453411262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4015070 con 0x5558aba00070 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffaa77fe640 1 -- 172.21.15.28:0/3453411262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab4004ec0 con 0x5558aba00070 2024-09-21T05:47:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.681+0000 7ffaa77fe640 1 -- 172.21.15.28:0/3453411262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab40308f0 con 0x5558aba00070 2024-09-21T05:47:59.687 INFO:tasks.workunit.client.0.smithi028.stdout: /bin/sh ./config.status 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 -- 172.21.15.28:0/3453411262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 msgr2=0x5558abdbed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ffab40118c0 tx=0x7ffab4005cf0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 -- 172.21.15.28:0/3453411262 shutdown_connections 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558aba00070 0x5558abdbed00 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558ab9df390 0x5558abdb4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 --2- 172.21.15.28:0/3453411262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9d71b0 0x5558abda8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 -- 172.21.15.28:0/3453411262 >> 172.21.15.28:0/3453411262 conn(0x5558ab9f4460 msgr2=0x5558abdb3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 -- 172.21.15.28:0/3453411262 shutdown_connections 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.682+0000 7ffaca277740 1 -- 172.21.15.28:0/3453411262 wait complete. 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 -- start start 2024-09-21T05:47:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 Processor -- start 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abe23760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abe2e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1b0cb0 con 0x5558ab9d71b0 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1b05e0 con 0x5558aba00070 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.683+0000 7ffaca277740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1ad820 con 0x5558ab9df390 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffaca277740 10 client.0 _ll_register_callbacks cb 0x7ffab400bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abe23760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac22a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35834/0 (socket says 172.21.15.28:35834) 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac22a0640 1 -- 172.21.15.28:0/2209009795 learned_addr learned my addr 172.21.15.28:0/2209009795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abe2e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 -- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 msgr2=0x5558abe2e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abe2e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 -- 172.21.15.28:0/2209009795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 msgr2=0x5558abe23220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 -- 172.21.15.28:0/2209009795 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558abe24050 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffac129e640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abe23760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffaac02a930 tx=0x7ffaac005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc7b974efa32d14 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.684+0000 7ffaa57fa640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac035040 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ffaac005040 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab4004ec0 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffaac005240 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 10 client.25753 fetch_fsmap learned FSMap version 27 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558abe2f8f0 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab4004ec0 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffaac053020 con 0x5558ab9df390 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 10 client.25753 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:47:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 10 client.25753 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 20 client.25753 populate_metadata read hostname 'smithi028' 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab993da0 0x5558abe32090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 4 client.25753 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.685+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558abd1b8a0 con 0x5558ab993da0 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab993da0 0x5558abe32090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffac22a0640 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab993da0 0x5558abe32090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a2c8cd164f003e86 server_cookie=bb99be0de6b7fb8 in_seq=0 out_seq=0 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaa57fa640 10 client.25753 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaa57fa640 1 -- 172.21.15.28:0/2209009795 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558abd1b8a0 con 0x5558ab993da0 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaa57fa640 10 client.25753 handle_command_reply: tid=0 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaca277740 1 client.25753 shutdown 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 msgr2=0x5558abe23760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.686+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abe23760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffaac02a930 tx=0x7ffaac005db0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 shutdown_connections 2024-09-21T05:47:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5558ab993da0 0x5558abe32090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558aba00070 0x5558abe2e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5558ab9df390 0x5558abe23760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaa57fa640 0 client.25753 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 --2- 172.21.15.28:0/2209009795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558ab9d71b0 0x5558abe23220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 >> 172.21.15.28:0/2209009795 conn(0x5558ab9f4460 msgr2=0x5558abda8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 shutdown_connections 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 1 -- 172.21.15.28:0/2209009795 wait complete. 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffaca277740 20 client.25753 trim_cache size 0 max 16384 2024-09-21T05:47:59.694 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 msgr2=0x7ffaa80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 0x7ffaa80b8c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5558abda5f40 tx=0x7ffab8077870 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 msgr2=0x7ffabc19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.687+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ffab800ebc0 tx=0x7ffab800c6a0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 shutdown_connections 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffabc10ddf0 0x7ffabc19fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffabc105270 0x7ffabc19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ffaa80b6700 0x7ffaa80b8c10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 --2- 172.21.15.28:0/2176290179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc104840 0x7ffabc19a520 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:47:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 >> 172.21.15.28:0/2176290179 conn(0x7ffabc079730 msgr2=0x7ffabc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:47:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 shutdown_connections 2024-09-21T05:47:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:47:59.688+0000 7ffac3d03640 1 -- 172.21.15.28:0/2176290179 wait complete. 2024-09-21T05:47:59.737 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-09-21T05:47:59.751 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-21T05:47:59.775 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-09-21T05:47:59.786 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-09-21T05:47:59.814 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-21T05:48:00.004 INFO:tasks.workunit.client.0.smithi028.stdout:rm -f stamp-h1 2024-09-21T05:48:00.007 INFO:tasks.workunit.client.0.smithi028.stdout:touch config.h.in 2024-09-21T05:48:00.009 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh ./config.status config.h 2024-09-21T05:48:00.051 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-21T05:48:00.075 INFO:tasks.workunit.client.0.smithi028.stdout:make all-am 2024-09-21T05:48:00.081 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-21T05:48:00.081 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-21T05:48:00.341 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-21T05:48:00.344 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-21T05:48:00.513 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-21T05:48:00.516 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-21T05:48:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:00 smithi028 ceph-mon[31227]: pgmap v354: 97 pgs: 97 active+clean; 6.1 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 450 KiB/s wr, 63 op/s 2024-09-21T05:48:00.608 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-21T05:48:00.612 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-21T05:48:00.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:48:00.701 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-21T05:48:00.704 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-21T05:48:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:00 smithi181 ceph-mon[34819]: pgmap v354: 97 pgs: 97 active+clean; 6.1 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 450 KiB/s wr, 63 op/s 2024-09-21T05:48:00.831 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-21T05:48:00.834 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-21T05:48:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:00 smithi129 ceph-mon[34885]: pgmap v354: 97 pgs: 97 active+clean; 6.1 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 450 KiB/s wr, 63 op/s 2024-09-21T05:48:00.885 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-21T05:48:00.888 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-21T05:48:00.926 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:00.933 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-21T05:48:00.936 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-21T05:48:00.989 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-21T05:48:00.995 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-21T05:48:01.213 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-21T05:48:01.216 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-21T05:48:01.358 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-21T05:48:01.361 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 -- 172.21.15.28:0/3304095238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb41083a0 msgr2=0x7fceb41087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 --2- 172.21.15.28:0/3304095238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb41083a0 0x7fceb41087b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009900 tx=0x7fce9c02a410 comp rx=0 tx=0).stop 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 -- 172.21.15.28:0/3304095238 shutdown_connections 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 --2- 172.21.15.28:0/3304095238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb4108cf0 0x7fceb410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 --2- 172.21.15.28:0/3304095238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb41083a0 0x7fceb41087b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 --2- 172.21.15.28:0/3304095238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4102a60 0x7fceb4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 -- 172.21.15.28:0/3304095238 >> 172.21.15.28:0/3304095238 conn(0x7fceb4079730 msgr2=0x7fceb4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 -- 172.21.15.28:0/3304095238 shutdown_connections 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.439+0000 7fceba14c640 1 -- 172.21.15.28:0/3304095238 wait complete. 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.440+0000 7fceba14c640 1 Processor -- start 2024-09-21T05:48:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.440+0000 7fceba14c640 1 -- start start 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.440+0000 7fceba14c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceba14c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb41083a0 0x7fceb419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceba14c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4108cf0 0x7fceb41a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceba14c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4112960 con 0x7fceb4102a60 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceba14c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41127e0 con 0x7fceb4108cf0 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceba14c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fceb4112ae0 con 0x7fceb41083a0 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35846/0 (socket says 172.21.15.28:35846) 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 -- 172.21.15.28:0/3447246036 learned_addr learned my addr 172.21.15.28:0/3447246036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceabfff640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb41083a0 0x7fceb419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb3fff640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4108cf0 0x7fceb41a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 -- 172.21.15.28:0/3447246036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb41083a0 msgr2=0x7fceb419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb41083a0 0x7fceb419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 -- 172.21.15.28:0/3447246036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4108cf0 msgr2=0x7fceb41a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4108cf0 0x7fceb41a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 -- 172.21.15.28:0/3447246036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb41a24f0 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb37fe640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcea400c920 tx=0x7fcea400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb17fa640 1 -- 172.21.15.28:0/3447246036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4007c70 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.441+0000 7fceb17fa640 1 -- 172.21.15.28:0/3447246036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea4007e10 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.442+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb41a27e0 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.442+0000 7fceb17fa640 1 -- 172.21.15.28:0/3447246036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4018820 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.442+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb41a9000 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.444+0000 7fceb17fa640 1 -- 172.21.15.28:0/3447246036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fcea400f450 con 0x7fceb4102a60 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.445+0000 7fcec06c0740 1 Processor -- start 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.445+0000 7fceb17fa640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 0x7fcea00b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.445+0000 7fcec06c0740 1 -- start start 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.445+0000 7fceb3fff640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 0x7fcea00b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.445+0000 7fceb17fa640 1 -- 172.21.15.28:0/3447246036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcea40e78f0 con 0x7fceb4102a60 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7ae26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ff7a58ea0 con 0x555ff77121b0 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555ff7a482d0 con 0x555ff771a390 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fcec06c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555ff7a4c0e0 con 0x555ff773b070 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35850/0 (socket says 172.21.15.28:35850) 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 -- 172.21.15.28:0/2265704399 learned_addr learned my addr 172.21.15.28:0/2265704399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceb3fff640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 0x7fcea00b8bc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fceac00ad60 tx=0x7fceac00a2f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceb37fe640 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7ae26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceb3fff640 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 -- 172.21.15.28:0/2265704399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 msgr2=0x555ff7ae26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7ae26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 -- 172.21.15.28:0/2265704399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 msgr2=0x555ff7ae3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 -- 172.21.15.28:0/2265704399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ff77a6650 con 0x555ff77121b0 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.446+0000 7fceabfff640 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fceb406aaa0 tx=0x7fce9c031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce09800c6b48ccf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.447+0000 7fcea9ffb640 1 -- 172.21.15.28:0/2265704399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c03f070 con 0x555ff77121b0 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.447+0000 7fcea9ffb640 1 -- 172.21.15.28:0/2265704399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9c0316a0 con 0x555ff77121b0 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.447+0000 7fcea9ffb640 1 -- 172.21.15.28:0/2265704399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c035070 con 0x555ff77121b0 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- 172.21.15.28:0/2265704399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 msgr2=0x555ff7ae2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fceb406aaa0 tx=0x7fce9c031cd0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- 172.21.15.28:0/2265704399 shutdown_connections 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7ae2be0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 --2- 172.21.15.28:0/2265704399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7ae26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- 172.21.15.28:0/2265704399 >> 172.21.15.28:0/2265704399 conn(0x555ff772f460 msgr2=0x555ff7ae0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- 172.21.15.28:0/2265704399 shutdown_connections 2024-09-21T05:48:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- 172.21.15.28:0/2265704399 wait complete. 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.448+0000 7fcec06c0740 1 -- start start 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 Processor -- start 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff77121b0 0x555ff7aedd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ff78a9700 con 0x555ff773b070 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555ff79e3160 con 0x555ff77121b0 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555ff7af0e20 con 0x555ff771a390 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fcec06c0740 10 client.0 _ll_register_callbacks cb 0x555ff79fc9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fceabfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fceabfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35010/0 (socket says 172.21.15.28:35010) 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fceabfff640 1 -- 172.21.15.28:0/4018945180 learned_addr learned my addr 172.21.15.28:0/4018945180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.449+0000 7fceb37fe640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff77121b0 0x555ff7aedd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 -- 172.21.15.28:0/4018945180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff77121b0 msgr2=0x555ff7aedd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff77121b0 0x555ff7aedd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 -- 172.21.15.28:0/4018945180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 msgr2=0x555ff7afa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 -- 172.21.15.28:0/4018945180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ff7a10c90 con 0x555ff771a390 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceb3fff640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fceabfff640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fce9c02a450 tx=0x7fce9c031570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2601114b08290af6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.450+0000 7fce92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.453+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c04c060 con 0x555ff771a390 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fce9c004230 con 0x555ff771a390 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce9c0316a0 con 0x555ff771a390 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fce9c035070 con 0x555ff771a390 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 10 client.25762 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555ff7afcb30 con 0x555ff771a390 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9c0316a0 con 0x555ff771a390 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fce9c0037b0 con 0x555ff771a390 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 10 client.25762 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 10 client.25762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 20 client.25762 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76f99f0 0x555ff7affe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 4 client.25762 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.454+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ff7a5adf0 con 0x555ff76f99f0 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.455+0000 7fceb37fe640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76f99f0 0x555ff7affe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.455+0000 7fce92ffd640 10 client.25762 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.455+0000 7fceb37fe640 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76f99f0 0x555ff7affe90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4a76935833adf26a server_cookie=4d2d3fe7f6c986d3 in_seq=0 out_seq=0 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fce92ffd640 1 -- 172.21.15.28:0/4018945180 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555ff7a5adf0 con 0x555ff76f99f0 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fce92ffd640 10 client.25762 handle_command_reply: tid=0 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 client.25762 shutdown 2024-09-21T05:48:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 msgr2=0x555ff7aee2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fce9c02a450 tx=0x7fce9c031570 comp rx=0 tx=0).stop 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 shutdown_connections 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76f99f0 0x555ff7affe90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff771a390 0x555ff7aee2d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 --2- 172.21.15.28:0/4018945180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff77121b0 0x555ff7aedd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fce92ffd640 0 client.25762 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 >> 172.21.15.28:0/4018945180 conn(0x555ff772f460 msgr2=0x555ff7ae5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 shutdown_connections 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.456+0000 7fcec06c0740 1 -- 172.21.15.28:0/4018945180 wait complete. 2024-09-21T05:48:01.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.457+0000 7fcec06c0740 20 client.25762 trim_cache size 0 max 16384 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 Processor -- start 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 -- start start 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff77121b0 0x555ff7aefc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41b1720 con 0x555ff773b070 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41b3260 con 0x555ff771a390 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.470+0000 7fcec06c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41b1df0 con 0x555ff77121b0 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35870/0 (socket says 172.21.15.28:35870) 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 -- 172.21.15.28:0/754324775 learned_addr learned my addr 172.21.15.28:0/754324775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceabfff640 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb37fe640 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff77121b0 0x555ff7aefc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 -- 172.21.15.28:0/754324775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff77121b0 msgr2=0x555ff7aefc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff77121b0 0x555ff7aefc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 -- 172.21.15.28:0/754324775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 msgr2=0x555ff7ae3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 -- 172.21.15.28:0/754324775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ff7ae0f00 con 0x555ff773b070 2024-09-21T05:48:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fceb3fff640 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fceac010900 tx=0x7fceac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b6f911105b21b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.471+0000 7fcea97fa640 1 -- 172.21.15.28:0/754324775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac01ebb0 con 0x555ff773b070 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcea97fa640 1 -- 172.21.15.28:0/754324775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac01ed50 con 0x555ff773b070 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcea97fa640 1 -- 172.21.15.28:0/754324775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac019690 con 0x555ff773b070 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 -- 172.21.15.28:0/754324775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 msgr2=0x555ff7afa0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fceac010900 tx=0x7fceac011000 comp rx=0 tx=0).stop 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 -- 172.21.15.28:0/754324775 shutdown_connections 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff773b070 0x555ff7afa0b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7ae3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.472+0000 7fcec06c0740 1 --2- 172.21.15.28:0/754324775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff77121b0 0x555ff7aefc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.473+0000 7fcec06c0740 1 -- 172.21.15.28:0/754324775 >> 172.21.15.28:0/754324775 conn(0x555ff772f460 msgr2=0x555ff7ae36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.473+0000 7fcec06c0740 1 -- 172.21.15.28:0/754324775 shutdown_connections 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.473+0000 7fcec06c0740 1 -- 172.21.15.28:0/754324775 wait complete. 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.473+0000 7fcec06c0740 1 -- start start 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.473+0000 7fcec06c0740 1 Processor -- start 2024-09-21T05:48:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7b013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7b018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41b2b90 con 0x555ff77121b0 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41b24c0 con 0x555ff771a390 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41af700 con 0x555ff773b070 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fcec06c0740 10 client.0 _ll_register_callbacks cb 0x7fcea400a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35882/0 (socket says 172.21.15.28:35882) 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 -- 172.21.15.28:0/252441112 learned_addr learned my addr 172.21.15.28:0/252441112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb3fff640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7b018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceabfff640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7b013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 -- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 msgr2=0x555ff7b018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7b018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 -- 172.21.15.28:0/252441112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 msgr2=0x555ff7b013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7b013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 -- 172.21.15.28:0/252441112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ff7b69610 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fceb37fe640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcea40e5110 tx=0x7fcea40a9db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f193a28460dda59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.474+0000 7fce92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea40a7e60 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcea40a99d0 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceac01ed50 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcea40b7180 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 10 client.16128 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555ff7b6a8a0 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea40b6cf0 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fcea40ac430 con 0x555ff77121b0 2024-09-21T05:48:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 10 client.16128 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 10 client.16128 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 20 client.16128 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76ceda0 0x555ff7b6d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 4 client.16128 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.475+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fceac01ed50 con 0x555ff76ceda0 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fceabfff640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76ceda0 0x555ff7b6d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fce92ffd640 10 client.16128 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fceabfff640 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76ceda0 0x555ff7b6d010 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fbfd5223f845ff05 server_cookie=9692bebb9e077ab1 in_seq=0 out_seq=0 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fce92ffd640 1 -- 172.21.15.28:0/252441112 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fceac01ed50 con 0x555ff76ceda0 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fce92ffd640 10 client.16128 handle_command_reply: tid=0 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.476+0000 7fcec06c0740 1 client.16128 shutdown 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 msgr2=0x555ff7b00e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcea40e5110 tx=0x7fcea40a9db0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 shutdown_connections 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x555ff76ceda0 0x555ff7b6d010 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ff773b070 0x555ff7b018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ff771a390 0x555ff7b013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 --2- 172.21.15.28:0/252441112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ff77121b0 0x555ff7b00e70 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 >> 172.21.15.28:0/252441112 conn(0x555ff772f460 msgr2=0x555ff7aefad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fce92ffd640 0 client.16128 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 shutdown_connections 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 1 -- 172.21.15.28:0/252441112 wait complete. 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.477+0000 7fcec06c0740 20 client.16128 trim_cache size 0 max 16384 2024-09-21T05:48:01.487 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 msgr2=0x7fcea00b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 0x7fcea00b8bc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fceac00ad60 tx=0x7fceac00a2f0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 msgr2=0x7fceb419c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcea400c920 tx=0x7fcea400cde0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 shutdown_connections 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fceb4108cf0 0x7fceb41a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fceb41083a0 0x7fceb419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcea00b66b0 0x7fcea00b8bc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 --2- 172.21.15.28:0/3447246036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb4102a60 0x7fceb419c640 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 >> 172.21.15.28:0/3447246036 conn(0x7fceb4079730 msgr2=0x7fceb4104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 shutdown_connections 2024-09-21T05:48:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:01.478+0000 7fceba14c640 1 -- 172.21.15.28:0/3447246036 wait complete. 2024-09-21T05:48:01.547 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-21T05:48:01.551 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-21T05:48:01.627 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-21T05:48:01.631 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-21T05:48:01.724 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-21T05:48:01.728 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-21T05:48:01.846 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-21T05:48:01.849 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from parser.h:22, 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr: from parser.c:25: 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_filesys’: 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.879 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.880 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.881 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.882 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-21T05:48:01.883 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_tg_stats’: 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.884 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.885 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.886 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-21T05:48:01.887 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-21T05:48:01.888 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.930 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘get_optstr’: 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:200:14: note: source object declared here 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr: 200 | char temp[BUFSIZE]; 2024-09-21T05:48:01.931 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-09-21T05:48:01.947 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘size64_convert’: 2024-09-21T05:48:01.947 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-21T05:48:01.947 INFO:tasks.workunit.client.0.smithi028.stderr: 119 | memcpy(unit, "\0", 3); 2024-09-21T05:48:01.947 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-09-21T05:48:02.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:48:02.278 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-21T05:48:02.281 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-21T05:48:02.340 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-21T05:48:02.343 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-21T05:48:02.435 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-21T05:48:02.439 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-21T05:48:02.472 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-21T05:48:02.476 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-21T05:48:02.511 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-21T05:48:02.512 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-09-21T05:48:02.512 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir tmp 2024-09-21T05:48:02.514 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd tmp 2024-09-21T05:48:02.514 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in $mydir/*.ffsb 2024-09-21T05:48:02.514 INFO:tasks.workunit.client.0.smithi028.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-21T05:48:02.514 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB version 6.0-RC2 started 2024-09-21T05:48:02.515 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout:benchmark time = 300 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout:ThreadGroup 0 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout:================ 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: num_threads = 32 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: read_random = off 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: read_size = 0 (0B) 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: read_blocksize = 0 (0B) 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: read_skip = off 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: read_skipsize = 0 (0B) 2024-09-21T05:48:02.518 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: write_random = on 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: write_size = 5242880 (5MB) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: fsync_file = 0 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: write_blocksize = 4096 (4KB) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: wait time = 0 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: op weights 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: read = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: readall = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: write = 1 (100.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: create = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: append = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: delete = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: metaop = 0 (0.00%) 2024-09-21T05:48:02.519 INFO:tasks.workunit.client.0.smithi028.stdout: createdir = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: stat = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: writeall = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: writeall_fsync = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: open_close = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: write_fsync = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: create_fsync = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: append_fsync = 0 (0.00%) 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout:FileSystem . 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout:========== 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: num_dirs = 0 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: starting files = 128 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.520 INFO:tasks.workunit.client.0.smithi028.stdout: min file size = 104857600 (100MB) 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: max file size = 104857600 (100MB) 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: directio = off 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: alignedio = on 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: bufferedio = off 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: aging is off 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: current utilization = 0.00% 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout:checking existing fs: . 2024-09-21T05:48:02.521 INFO:tasks.workunit.client.0.smithi028.stdout:recreating new fileset 2024-09-21T05:48:02.522 INFO:tasks.workunit.client.0.smithi028.stderr:opendir: No such file or directory 2024-09-21T05:48:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:02 smithi028 ceph-mon[31227]: pgmap v355: 97 pgs: 97 active+clean; 6.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 476 KiB/s wr, 65 op/s 2024-09-21T05:48:02.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:02 smithi181 ceph-mon[34819]: pgmap v355: 97 pgs: 97 active+clean; 6.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 476 KiB/s wr, 65 op/s 2024-09-21T05:48:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:02 smithi129 ceph-mon[34885]: pgmap v355: 97 pgs: 97 active+clean; 6.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 476 KiB/s wr, 65 op/s 2024-09-21T05:48:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 -- 172.21.15.28:0/3539246047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9464109df0 msgr2=0x7f9464110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 --2- 172.21.15.28:0/3539246047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9464109df0 0x7f9464110620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f945c009ca0 tx=0x7f945c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 -- 172.21.15.28:0/3539246047 shutdown_connections 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 --2- 172.21.15.28:0/3539246047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9464109df0 0x7f9464110620 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 --2- 172.21.15.28:0/3539246047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94641094a0 0x7f94641098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 --2- 172.21.15.28:0/3539246047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464103b60 0x7f9464103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 -- 172.21.15.28:0/3539246047 >> 172.21.15.28:0/3539246047 conn(0x7f946407b850 msgr2=0x7f946407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 -- 172.21.15.28:0/3539246047 shutdown_connections 2024-09-21T05:48:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.007+0000 7f946991e640 1 -- 172.21.15.28:0/3539246047 wait complete. 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.008+0000 7f946991e640 1 Processor -- start 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 -- start start 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9464103b60 0x7f946419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464109df0 0x7f946419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9464113ab0 con 0x7f94641094a0 2024-09-21T05:48:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9464113930 con 0x7f9464109df0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f946991e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9464113c30 con 0x7f9464103b60 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35902/0 (socket says 172.21.15.28:35902) 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f94627fc640 1 -- 172.21.15.28:0/867834109 learned_addr learned my addr 172.21.15.28:0/867834109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f94637fe640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464109df0 0x7f946419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.009+0000 7f9462ffd640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9464103b60 0x7f946419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 -- 172.21.15.28:0/867834109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9464103b60 msgr2=0x7f946419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9464103b60 0x7f946419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 -- 172.21.15.28:0/867834109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464109df0 msgr2=0x7f946419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464109df0 0x7f946419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 -- 172.21.15.28:0/867834109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94641a03d0 con 0x7f94641094a0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f94627fc640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f945800d750 tx=0x7f945800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f946891c640 1 -- 172.21.15.28:0/867834109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458017070 con 0x7f94641094a0 2024-09-21T05:48:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f946891c640 1 -- 172.21.15.28:0/867834109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94580054d0 con 0x7f94641094a0 2024-09-21T05:48:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f946891c640 1 -- 172.21.15.28:0/867834109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458004c30 con 0x7f94641094a0 2024-09-21T05:48:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94641a06c0 con 0x7f94641094a0 2024-09-21T05:48:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.010+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94641a6e20 con 0x7f94641094a0 2024-09-21T05:48:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 Processor -- start 2024-09-21T05:48:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 -- start start 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6984be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d6985120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649d68faea0 con 0x5649d65dd070 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649d68ea2d0 con 0x5649d65b41b0 2024-09-21T05:48:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f946fe92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649d68ee0e0 con 0x5649d65bc390 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f94637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.013+0000 7f94637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35906/0 (socket says 172.21.15.28:35906) 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 -- 172.21.15.28:0/2072701820 learned_addr learned my addr 172.21.15.28:0/2072701820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f9462ffd640 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6984be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 -- 172.21.15.28:0/2072701820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 msgr2=0x5649d6985120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d6985120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 -- 172.21.15.28:0/2072701820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 msgr2=0x5649d6984be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6984be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 -- 172.21.15.28:0/2072701820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d6648650 con 0x5649d65dd070 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f94637fe640 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f945c0098b0 tx=0x7f945c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21179a7824deb3ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f9449ffb640 1 -- 172.21.15.28:0/2072701820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c031cf0 con 0x5649d65dd070 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f9449ffb640 1 -- 172.21.15.28:0/2072701820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f945c036030 con 0x5649d65dd070 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f9449ffb640 1 -- 172.21.15.28:0/2072701820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c004260 con 0x5649d65dd070 2024-09-21T05:48:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.014+0000 7f946891c640 1 -- 172.21.15.28:0/867834109 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9458004dd0 con 0x7f94641094a0 2024-09-21T05:48:03.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.015+0000 7f946891c640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 0x7f944c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.015+0000 7f946fe92740 1 -- 172.21.15.28:0/2072701820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 msgr2=0x5649d69846a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.015+0000 7f946fe92740 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f945c0098b0 tx=0x7f945c0098e0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.015+0000 7f9462ffd640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 0x7f944c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.015+0000 7f946891c640 1 -- 172.21.15.28:0/867834109 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f94580e6ff0 con 0x7f94641094a0 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 -- 172.21.15.28:0/2072701820 shutdown_connections 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d6985120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6984be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 --2- 172.21.15.28:0/2072701820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d69846a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 -- 172.21.15.28:0/2072701820 >> 172.21.15.28:0/2072701820 conn(0x5649d65d1460 msgr2=0x5649d6982590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 -- 172.21.15.28:0/2072701820 shutdown_connections 2024-09-21T05:48:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 -- 172.21.15.28:0/2072701820 wait complete. 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f9462ffd640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 0x7f944c0b8c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5649d6983320 tx=0x7f945400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.016+0000 7f946fe92740 1 -- start start 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 Processor -- start 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d698fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d69902a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d699bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649d674b700 con 0x5649d65dd070 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649d6885160 con 0x5649d65b41b0 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f946fe92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649d6992ca0 con 0x5649d65bc390 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f94637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d699bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.017+0000 7f9462ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d698fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f94637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d699bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35914/0 (socket says 172.21.15.28:35914) 2024-09-21T05:48:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f94637fe640 1 -- 172.21.15.28:0/8850044 learned_addr learned my addr 172.21.15.28:0/8850044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f946fe92740 10 client.0 _ll_register_callbacks cb 0x5649d689e9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f94627fc640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d69902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 -- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 msgr2=0x5649d69902a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d69902a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 -- 172.21.15.28:0/8850044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 msgr2=0x5649d699bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d699bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 -- 172.21.15.28:0/8850044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d68b2c90 con 0x5649d65b41b0 2024-09-21T05:48:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f94627fc640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d69902a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9462ffd640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d698fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f94540039c0 tx=0x7f9454004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc8442722c3fa1c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9442ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945402c060 con 0x5649d65b41b0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.018+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9454017030 con 0x5649d65b41b0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f945c036030 con 0x5649d65b41b0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f945401fd10 con 0x5649d65b41b0 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 10 client.25730 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649d699e9c0 con 0x5649d65b41b0 2024-09-21T05:48:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f945c036030 con 0x5649d65b41b0 2024-09-21T05:48:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f945401c430 con 0x5649d65b41b0 2024-09-21T05:48:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 10 client.25730 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 10 client.25730 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 20 client.25730 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d659b9f0 0x5649d69a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 4 client.25730 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.019+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649d68fcdf0 con 0x5649d659b9f0 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.020+0000 7f94627fc640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d659b9f0 0x5649d69a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.020+0000 7f9442ffd640 10 client.25730 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.020+0000 7f94627fc640 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d659b9f0 0x5649d69a1f20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0e591fc7c12d912 server_cookie=97c23d6c4396ae2 in_seq=0 out_seq=0 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f9442ffd640 1 -- 172.21.15.28:0/8850044 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5649d68fcdf0 con 0x5649d659b9f0 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f9442ffd640 10 client.25730 handle_command_reply: tid=0 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 client.25730 shutdown 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 msgr2=0x5649d698fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d698fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f94540039c0 tx=0x7f9454004730 comp rx=0 tx=0).stop 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 shutdown_connections 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d659b9f0 0x5649d69a1f20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65dd070 0x5649d699bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65bc390 0x5649d69902a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f9442ffd640 0 client.25730 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 --2- 172.21.15.28:0/8850044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d698fd60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 >> 172.21.15.28:0/8850044 conn(0x5649d65d1460 msgr2=0x5649d6987880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 shutdown_connections 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.021+0000 7f946fe92740 1 -- 172.21.15.28:0/8850044 wait complete. 2024-09-21T05:48:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.022+0000 7f946fe92740 20 client.25730 trim_cache size 0 max 16384 2024-09-21T05:48:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 Processor -- start 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 -- start start 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 0x5649d6981bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94641af6b0 con 0x5649d65bc390 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b11f0 con 0x5649d65dd070 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.035+0000 7f946fe92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94641afd80 con 0x5649d65b41b0 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f9462ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94637fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 0x5649d6981bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35922/0 (socket says 172.21.15.28:35922) 2024-09-21T05:48:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f9462ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35066/0 (socket says 172.21.15.28:35066) 2024-09-21T05:48:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 -- 172.21.15.28:0/1224137861 learned_addr learned my addr 172.21.15.28:0/1224137861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 -- 172.21.15.28:0/1224137861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 msgr2=0x5649d698ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 -- 172.21.15.28:0/1224137861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 msgr2=0x5649d6981bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 0x5649d6981bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 -- 172.21.15.28:0/1224137861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d698e260 con 0x5649d65bc390 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f9462ffd640 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94637fe640 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 0x5649d6981bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.036+0000 7f94627fc640 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9458009f40 tx=0x7f9458009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e8e5b370de47eb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.037+0000 7f94497fa640 1 -- 172.21.15.28:0/1224137861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458020040 con 0x5649d65bc390 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.037+0000 7f94497fa640 1 -- 172.21.15.28:0/1224137861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9458005680 con 0x5649d65bc390 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.037+0000 7f94497fa640 1 -- 172.21.15.28:0/1224137861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94580d4430 con 0x5649d65bc390 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.037+0000 7f946fe92740 1 -- 172.21.15.28:0/1224137861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 msgr2=0x5649d6985de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.037+0000 7f946fe92740 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9458009f40 tx=0x7f9458009f90 comp rx=0 tx=0).stop 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 -- 172.21.15.28:0/1224137861 shutdown_connections 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65dd070 0x5649d6981bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6985de0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 --2- 172.21.15.28:0/1224137861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65b41b0 0x5649d698ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 -- 172.21.15.28:0/1224137861 >> 172.21.15.28:0/1224137861 conn(0x5649d65d1460 msgr2=0x5649d69856d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 -- 172.21.15.28:0/1224137861 shutdown_connections 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 -- 172.21.15.28:0/1224137861 wait complete. 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 -- start start 2024-09-21T05:48:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.038+0000 7f946fe92740 1 Processor -- start 2024-09-21T05:48:03.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6998890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65dd070 0x5649d6994430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b0b20 con 0x5649d65bc390 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b0450 con 0x5649d65b41b0 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94641ad690 con 0x5649d65dd070 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f946fe92740 10 client.0 _ll_register_callbacks cb 0x7f9454004260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f9462ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6998890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35924/0 (socket says 172.21.15.28:35924) 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 -- 172.21.15.28:0/2874217610 learned_addr learned my addr 172.21.15.28:0/2874217610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94637fe640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65dd070 0x5649d6994430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 -- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65dd070 msgr2=0x5649d6994430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65dd070 0x5649d6994430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 -- 172.21.15.28:0/2874217610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 msgr2=0x5649d6998890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6998890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 -- 172.21.15.28:0/2874217610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d6a0fd30 con 0x5649d65bc390 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f9462ffd640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6998890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f94627fc640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94580a94e0 tx=0x7f94580e4c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc20e6a1578ab10f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.039+0000 7f9442ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458020070 con 0x5649d65bc390 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9458098070 con 0x5649d65bc390 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9458005680 con 0x5649d65bc390 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9458098270 con 0x5649d65bc390 2024-09-21T05:48:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 10 client.16146 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649d6a10e00 con 0x5649d65bc390 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9458005680 con 0x5649d65bc390 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9458092ce0 con 0x5649d65bc390 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 10 client.16146 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 10 client.16146 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 20 client.16146 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d6570da0 0x5649d6a13530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 4 client.16146 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.040+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649d68f88a0 con 0x5649d6570da0 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f9462ffd640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d6570da0 0x5649d6a13530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f9462ffd640 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d6570da0 0x5649d6a13530 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b2420a5f5268a225 server_cookie=cbdd8650c9c7665f in_seq=0 out_seq=0 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f9442ffd640 10 client.16146 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f9442ffd640 1 -- 172.21.15.28:0/2874217610 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649d68f88a0 con 0x5649d6570da0 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f9442ffd640 10 client.16146 handle_command_reply: tid=0 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.041+0000 7f946fe92740 1 client.16146 shutdown 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 msgr2=0x5649d6993d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94580a94e0 tx=0x7f94580e4c90 comp rx=0 tx=0).stop 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 shutdown_connections 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5649d6570da0 0x5649d6a13530 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649d65dd070 0x5649d6994430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649d65bc390 0x5649d6993d40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 --2- 172.21.15.28:0/2874217610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d65b41b0 0x5649d6998890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 >> 172.21.15.28:0/2874217610 conn(0x5649d65d1460 msgr2=0x5649d69911b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f9442ffd640 0 client.16146 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 shutdown_connections 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 1 -- 172.21.15.28:0/2874217610 wait complete. 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946fe92740 20 client.16146 trim_cache size 0 max 16384 2024-09-21T05:48:03.050 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.042+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 msgr2=0x7f944c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 0x7f944c0b8c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5649d6983320 tx=0x7f945400b390 comp rx=0 tx=0).stop 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 msgr2=0x7f946419aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f945800d750 tx=0x7f945800dc10 comp rx=0 tx=0).stop 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 shutdown_connections 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9464109df0 0x7f946419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f944c0b6700 0x7f944c0b8c10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94641094a0 0x7f946419aa00 secure :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f945800d750 tx=0x7f945800dc10 comp rx=0 tx=0).stop 2024-09-21T05:48:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 --2- 172.21.15.28:0/867834109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9464103b60 0x7f946419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 >> 172.21.15.28:0/867834109 conn(0x7f946407b850 msgr2=0x7f94641074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 shutdown_connections 2024-09-21T05:48:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:03.043+0000 7f946991e640 1 -- 172.21.15.28:0/867834109 wait complete. 2024-09-21T05:48:03.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:48:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:04.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:04 smithi028 ceph-mon[31227]: pgmap v356: 97 pgs: 97 active+clean; 3.4 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 621 KiB/s wr, 151 op/s 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 -- 172.21.15.28:0/3178903858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c1083a0 msgr2=0x7f173c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 --2- 172.21.15.28:0/3178903858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c1083a0 0x7f173c1087b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1734009be0 tx=0x7f173402a6d0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 -- 172.21.15.28:0/3178903858 shutdown_connections 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 --2- 172.21.15.28:0/3178903858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173c108cf0 0x7f173c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 --2- 172.21.15.28:0/3178903858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c1083a0 0x7f173c1087b0 secure :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1734009be0 tx=0x7f173402a6d0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 --2- 172.21.15.28:0/3178903858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f173c102a60 0x7f173c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.733+0000 7f1742c41640 1 -- 172.21.15.28:0/3178903858 >> 172.21.15.28:0/3178903858 conn(0x7f173c079730 msgr2=0x7f173c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.734+0000 7f1742c41640 1 -- 172.21.15.28:0/3178903858 shutdown_connections 2024-09-21T05:48:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.734+0000 7f1742c41640 1 -- 172.21.15.28:0/3178903858 wait complete. 2024-09-21T05:48:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.736+0000 7f1742c41640 1 Processor -- start 2024-09-21T05:48:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.736+0000 7f1742c41640 1 -- start start 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173c102a60 0x7f173c1a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f173c1a63e0 0x7f173c1a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f173c111860 con 0x7f173c108cf0 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1116e0 con 0x7f173c1a63e0 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.737+0000 7f1742c41640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1119e0 con 0x7f173c102a60 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35944/0 (socket says 172.21.15.28:35944) 2024-09-21T05:48:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 -- 172.21.15.28:0/1110344239 learned_addr learned my addr 172.21.15.28:0/1110344239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 -- 172.21.15.28:0/1110344239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173c102a60 msgr2=0x7f173c1a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173c102a60 0x7f173c1a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 -- 172.21.15.28:0/1110344239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f173c1a63e0 msgr2=0x7f173c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f173c1a63e0 0x7f173c1a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.738+0000 7f173bfff640 1 -- 172.21.15.28:0/1110344239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173c1a6fc0 con 0x7f173c108cf0 2024-09-21T05:48:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.739+0000 7f173bfff640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f173402acc0 tx=0x7f1734030640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.739+0000 7f1739ffb640 1 -- 172.21.15.28:0/1110344239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734041020 con 0x7f173c108cf0 2024-09-21T05:48:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.740+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173c19a0b0 con 0x7f173c108cf0 2024-09-21T05:48:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.740+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173c19a5b0 con 0x7f173c108cf0 2024-09-21T05:48:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.740+0000 7f1739ffb640 1 -- 172.21.15.28:0/1110344239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1734030e20 con 0x7f173c108cf0 2024-09-21T05:48:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.741+0000 7f1739ffb640 1 -- 172.21.15.28:0/1110344239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734036040 con 0x7f173c108cf0 2024-09-21T05:48:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.744+0000 7f17491b5740 1 Processor -- start 2024-09-21T05:48:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.744+0000 7f17491b5740 1 -- start start 2024-09-21T05:48:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.744+0000 7f17491b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf28070 0x55e39c2cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.744+0000 7f17491b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17491b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17491b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e39c245ea0 con 0x55e39beff1b0 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17491b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e39c2352d0 con 0x55e39bf28070 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17491b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e39c2390e0 con 0x55e39bf07390 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17409dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17409dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35958/0 (socket says 172.21.15.28:35958) 2024-09-21T05:48:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17409dd640 1 -- 172.21.15.28:0/223357820 learned_addr learned my addr 172.21.15.28:0/223357820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.745+0000 7f17411de640 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf28070 0x55e39c2cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 -- 172.21.15.28:0/223357820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 msgr2=0x55e39c2d0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:48:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2d0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 -- 172.21.15.28:0/223357820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf28070 msgr2=0x55e39c2cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf28070 0x55e39c2cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 -- 172.21.15.28:0/223357820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e39bf93650 con 0x55e39beff1b0 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f1739ffb640 1 -- 172.21.15.28:0/1110344239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1734044020 con 0x7f173c108cf0 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.746+0000 7f17409dd640 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f172c00bb10 tx=0x7f172c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae63022792bb95fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.747+0000 7f1722ffd640 1 -- 172.21.15.28:0/223357820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c016070 con 0x55e39beff1b0 2024-09-21T05:48:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.747+0000 7f1722ffd640 1 -- 172.21.15.28:0/223357820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f172c0054d0 con 0x55e39beff1b0 2024-09-21T05:48:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.747+0000 7f1722ffd640 1 -- 172.21.15.28:0/223357820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c004e60 con 0x55e39beff1b0 2024-09-21T05:48:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 -- 172.21.15.28:0/223357820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 msgr2=0x55e39c2cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f172c00bb10 tx=0x7f172c00bf10 comp rx=0 tx=0).stop 2024-09-21T05:48:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f1739ffb640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 0x7f17240b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 -- 172.21.15.28:0/223357820 shutdown_connections 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2cfbe0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 --2- 172.21.15.28:0/223357820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf28070 0x55e39c2cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.748+0000 7f17491b5740 1 -- 172.21.15.28:0/223357820 >> 172.21.15.28:0/223357820 conn(0x55e39bf1c460 msgr2=0x55e39c2cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.749+0000 7f17411de640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 0x7f17240b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.749+0000 7f17491b5740 1 -- 172.21.15.28:0/223357820 shutdown_connections 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.749+0000 7f17491b5740 1 -- 172.21.15.28:0/223357820 wait complete. 2024-09-21T05:48:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.749+0000 7f1739ffb640 1 -- 172.21.15.28:0/1110344239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1734107a20 con 0x7f173c108cf0 2024-09-21T05:48:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.749+0000 7f17411de640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 0x7f17240b8bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1730005ab0 tx=0x7f1730005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.750+0000 7f17491b5740 1 -- start start 2024-09-21T05:48:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.750+0000 7f17491b5740 1 Processor -- start 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39beff1b0 0x55e39c2dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c2db2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e39c096700 con 0x55e39bf28070 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e39c1d0160 con 0x55e39bf07390 2024-09-21T05:48:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e39c2dde80 con 0x55e39beff1b0 2024-09-21T05:48:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17411de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17411de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35970/0 (socket says 172.21.15.28:35970) 2024-09-21T05:48:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17411de640 1 -- 172.21.15.28:0/2221936036 learned_addr learned my addr 172.21.15.28:0/2221936036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.751+0000 7f17491b5740 10 client.0 _ll_register_callbacks cb 0x55e39c1e99e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 -- 172.21.15.28:0/2221936036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39beff1b0 msgr2=0x55e39c2dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39beff1b0 0x55e39c2dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 -- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 msgr2=0x55e39c2db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c2db2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 -- 172.21.15.28:0/2221936036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e39c1fdc90 con 0x55e39bf28070 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f17411de640 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f173000e880 tx=0x7f17300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760a6801723ea38 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.752+0000 7f171bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173000ee00 con 0x55e39bf28070 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f173000fad0 con 0x55e39bf28070 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172c0054d0 con 0x55e39bf28070 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f173000fcd0 con 0x55e39bf28070 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f17491b5740 10 client.16158 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.753+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e39c2e9cb0 con 0x55e39bf28070 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f173000fe70 con 0x55e39bf28070 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f173001e640 con 0x55e39bf28070 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 10 client.16158 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 10 client.16158 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 20 client.16158 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bee69f0 0x55e39c2ed220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 4 client.16158 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.754+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f172c0054d0 con 0x55e39bee69f0 2024-09-21T05:48:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.755+0000 7f17409dd640 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bee69f0 0x55e39c2ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.758+0000 7f17409dd640 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bee69f0 0x55e39c2ed220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26cbac46f008cc9e server_cookie=d01d57e32c77f694 in_seq=0 out_seq=0 2024-09-21T05:48:04.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.759+0000 7f171bfff640 10 client.16158 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f171bfff640 1 -- 172.21.15.28:0/2221936036 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f172c0054d0 con 0x55e39bee69f0 2024-09-21T05:48:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f171bfff640 10 client.16158 handle_command_reply: tid=0 2024-09-21T05:48:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 client.16158 shutdown 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 msgr2=0x55e39c2e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f173000e880 tx=0x7f17300046c0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 shutdown_connections 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bee69f0 0x55e39c2ed220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bf28070 0x55e39c2e7200 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f171bfff640 0 client.16158 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.760+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c2db2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f17491b5740 1 --2- 172.21.15.28:0/2221936036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39beff1b0 0x55e39c2dadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 >> 172.21.15.28:0/2221936036 conn(0x55e39bf1c460 msgr2=0x55e39c2d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 shutdown_connections 2024-09-21T05:48:04.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f17491b5740 1 -- 172.21.15.28:0/2221936036 wait complete. 2024-09-21T05:48:04.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.761+0000 7f17491b5740 20 client.16158 trim_cache size 0 max 16384 2024-09-21T05:48:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.783+0000 7f17491b5740 1 Processor -- start 2024-09-21T05:48:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.783+0000 7f17491b5740 1 -- start start 2024-09-21T05:48:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39beff1b0 0x55e39c2dc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f172800e8b0 con 0x55e39bee69f0 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17280103f0 con 0x55e39beff1b0 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17491b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f172800ef80 con 0x55e39bf07390 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17409dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17409dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35978/0 (socket says 172.21.15.28:35978) 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.784+0000 7f17409dd640 1 -- 172.21.15.28:0/191765344 learned_addr learned my addr 172.21.15.28:0/191765344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f173bfff640 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39beff1b0 0x55e39c2dc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f17409dd640 1 -- 172.21.15.28:0/191765344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 msgr2=0x55e39c2ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f17409dd640 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f17409dd640 1 -- 172.21.15.28:0/191765344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39beff1b0 msgr2=0x55e39c2dc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f17409dd640 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39beff1b0 0x55e39c2dc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.785+0000 7f17409dd640 1 -- 172.21.15.28:0/191765344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e39c2d92b0 con 0x55e39bee69f0 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.786+0000 7f17409dd640 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f172c000c00 tx=0x7f172c004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1ba9985bcccc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.786+0000 7f17227fc640 1 -- 172.21.15.28:0/191765344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c016040 con 0x55e39bee69f0 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.786+0000 7f17227fc640 1 -- 172.21.15.28:0/191765344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f172c027030 con 0x55e39bee69f0 2024-09-21T05:48:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.786+0000 7f17227fc640 1 -- 172.21.15.28:0/191765344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0038e0 con 0x55e39bee69f0 2024-09-21T05:48:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 -- 172.21.15.28:0/191765344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 msgr2=0x55e39c2d9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f172c000c00 tx=0x7f172c004a00 comp rx=0 tx=0).stop 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 -- 172.21.15.28:0/191765344 shutdown_connections 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bf07390 0x55e39c2ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39beff1b0 0x55e39c2dc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 --2- 172.21.15.28:0/191765344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39bee69f0 0x55e39c2d9ca0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 -- 172.21.15.28:0/191765344 >> 172.21.15.28:0/191765344 conn(0x55e39bf1c460 msgr2=0x55e39c2dc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 -- 172.21.15.28:0/191765344 shutdown_connections 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.787+0000 7f17491b5740 1 -- 172.21.15.28:0/191765344 wait complete. 2024-09-21T05:48:04.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.788+0000 7f17491b5740 1 -- start start 2024-09-21T05:48:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.788+0000 7f17491b5740 1 Processor -- start 2024-09-21T05:48:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2dfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c356c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f172800fd20 con 0x55e39beff1b0 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f172800f650 con 0x55e39bf07390 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f172800c890 con 0x55e39bee69f0 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35104/0 (socket says 172.21.15.28:35104) 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 -- 172.21.15.28:0/4258262855 learned_addr learned my addr 172.21.15.28:0/4258262855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17491b5740 10 client.0 _ll_register_callbacks cb 0x7f1730009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 -- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 msgr2=0x55e39c356c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c356c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 -- 172.21.15.28:0/4258262855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 msgr2=0x55e39c2dfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2dfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.789+0000 7f17409dd640 1 -- 172.21.15.28:0/4258262855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e39c357160 con 0x55e39bee69f0 2024-09-21T05:48:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f17409dd640 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f172c005cf0 tx=0x7f172c00dfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be70b6e2ba3ec40 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f171bfff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c016050 con 0x55e39bee69f0 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f172c00dae0 con 0x55e39bee69f0 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172c027030 con 0x55e39bee69f0 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f172c00dd00 con 0x55e39bee69f0 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f17491b5740 10 client.25789 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.790+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e39c358230 con 0x55e39bee69f0 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f172c027030 con 0x55e39bee69f0 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f172c003b50 con 0x55e39bee69f0 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 10 client.25789 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 10 client.25789 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 20 client.25789 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bf28070 0x55e39c35a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 4 client.25789 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e39c247df0 con 0x55e39bf28070 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.791+0000 7f173bfff640 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bf28070 0x55e39c35a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.792+0000 7f173bfff640 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bf28070 0x55e39c35a7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b67667fb5572cb4 server_cookie=d408088c66b5c818 in_seq=0 out_seq=0 2024-09-21T05:48:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.792+0000 7f171bfff640 10 client.25789 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:04.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:04 smithi181 ceph-mon[34819]: pgmap v356: 97 pgs: 97 active+clean; 3.4 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 621 KiB/s wr, 151 op/s 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f171bfff640 1 -- 172.21.15.28:0/4258262855 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e39c247df0 con 0x55e39bf28070 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f171bfff640 10 client.25789 handle_command_reply: tid=0 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 client.25789 shutdown 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 msgr2=0x55e39c2df4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f172c005cf0 tx=0x7f172c00dfa0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 shutdown_connections 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55e39bf28070 0x55e39c35a7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e39bf07390 0x55e39c356c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e39beff1b0 0x55e39c2dfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f171bfff640 0 client.25789 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 --2- 172.21.15.28:0/4258262855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e39bee69f0 0x55e39c2df4e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 >> 172.21.15.28:0/4258262855 conn(0x55e39bf1c460 msgr2=0x55e39c2db2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 shutdown_connections 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 1 -- 172.21.15.28:0/4258262855 wait complete. 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.793+0000 7f17491b5740 20 client.25789 trim_cache size 0 max 16384 2024-09-21T05:48:04.798 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 msgr2=0x7f17240b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 0x7f17240b8bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1730005ab0 tx=0x7f1730005a20 comp rx=0 tx=0).stop 2024-09-21T05:48:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 msgr2=0x7f173c1a1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f173402acc0 tx=0x7f1734030640 comp rx=0 tx=0).stop 2024-09-21T05:48:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 shutdown_connections 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f173c1a63e0 0x7f173c1a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17240b66e0 0x7f17240b8bf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f173c108cf0 0x7f173c1a1200 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 --2- 172.21.15.28:0/1110344239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173c102a60 0x7f173c1a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.794+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 >> 172.21.15.28:0/1110344239 conn(0x7f173c079730 msgr2=0x7f173c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.795+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 shutdown_connections 2024-09-21T05:48:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:04.795+0000 7f1742c41640 1 -- 172.21.15.28:0/1110344239 wait complete. 2024-09-21T05:48:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:04 smithi129 ceph-mon[34885]: pgmap v356: 97 pgs: 97 active+clean; 3.4 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 621 KiB/s wr, 151 op/s 2024-09-21T05:48:05.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:48:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:06.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.647+0000 7f51b77fe640 1 -- 172.21.15.28:0/957062390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac0318a0 con 0x7f51b81049b0 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.648+0000 7f51bef52640 1 -- 172.21.15.28:0/957062390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81049b0 msgr2=0x7f51b8104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.648+0000 7f51bef52640 1 --2- 172.21.15.28:0/957062390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81049b0 0x7f51b8104dc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f51ac009900 tx=0x7f51ac02a410 comp rx=0 tx=0).stop 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.648+0000 7f51bef52640 1 -- 172.21.15.28:0/957062390 shutdown_connections 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.648+0000 7f51bef52640 1 --2- 172.21.15.28:0/957062390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b810de10 0x7f51b8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.649+0000 7f51bef52640 1 --2- 172.21.15.28:0/957062390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b8105300 0x7f51b810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.649+0000 7f51bef52640 1 --2- 172.21.15.28:0/957062390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81049b0 0x7f51b8104dc0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.649+0000 7f51bef52640 1 -- 172.21.15.28:0/957062390 >> 172.21.15.28:0/957062390 conn(0x7f51b8079730 msgr2=0x7f51b8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.649+0000 7f51bef52640 1 -- 172.21.15.28:0/957062390 shutdown_connections 2024-09-21T05:48:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.649+0000 7f51bef52640 1 -- 172.21.15.28:0/957062390 wait complete. 2024-09-21T05:48:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.650+0000 7f51bef52640 1 Processor -- start 2024-09-21T05:48:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.650+0000 7f51bef52640 1 -- start start 2024-09-21T05:48:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b8105300 0x7f51b810a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 0x7f51b810ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8103620 con 0x7f51b81a6450 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81034a0 con 0x7f51b810de10 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bef52640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81037a0 con 0x7f51b8105300 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bd4ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bd4ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54012/0 (socket says 172.21.15.28:54012) 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bd4ef640 1 -- 172.21.15.28:0/2245168179 learned_addr learned my addr 172.21.15.28:0/2245168179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 0x7f51b810ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.651+0000 7f51bccee640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b8105300 0x7f51b810a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 -- 172.21.15.28:0/2245168179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b8105300 msgr2=0x7f51b810a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b8105300 0x7f51b810a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 -- 172.21.15.28:0/2245168179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 msgr2=0x7f51b810ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 0x7f51b810ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 -- 172.21.15.28:0/2245168179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b81a6fa0 con 0x7f51b81a6450 2024-09-21T05:48:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 0x7f51b810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.652+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f51b000cce0 tx=0x7f51b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.653+0000 7f51b5ffb640 1 -- 172.21.15.28:0/2245168179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b000f450 con 0x7f51b81a6450 2024-09-21T05:48:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.653+0000 7f51b5ffb640 1 -- 172.21.15.28:0/2245168179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b000faa0 con 0x7f51b81a6450 2024-09-21T05:48:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.653+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b81a7290 con 0x7f51b81a6450 2024-09-21T05:48:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.654+0000 7f51b5ffb640 1 -- 172.21.15.28:0/2245168179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0018b70 con 0x7f51b81a6450 2024-09-21T05:48:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.654+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b81ad830 con 0x7f51b81a6450 2024-09-21T05:48:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.658+0000 7f51b5ffb640 1 -- 172.21.15.28:0/2245168179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f51b0018d10 con 0x7f51b81a6450 2024-09-21T05:48:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.659+0000 7f51c54c6740 1 Processor -- start 2024-09-21T05:48:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.659+0000 7f51c54c6740 1 -- start start 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21b10070 0x563a21eb72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21aef390 0x563a21eb7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a21e2dea0 con 0x563a21aef390 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563a21e1d2d0 con 0x563a21ae71b0 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51c54c6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563a21e210e0 con 0x563a21b10070 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51b7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21b10070 0x563a21eb72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51bccee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51bccee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53818/0 (socket says 172.21.15.28:53818) 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.660+0000 7f51bccee640 1 -- 172.21.15.28:0/814263303 learned_addr learned my addr 172.21.15.28:0/814263303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 -- 172.21.15.28:0/814263303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21b10070 msgr2=0x563a21eb72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21b10070 0x563a21eb72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 -- 172.21.15.28:0/814263303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21aef390 msgr2=0x563a21eb7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21aef390 0x563a21eb7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 -- 172.21.15.28:0/814263303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a21b7b650 con 0x563a21ae71b0 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51b5ffb640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 0x7f51a00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51bccee640 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f51ac000c00 tx=0x7f51ac032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb4c053e8ac5f85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.661+0000 7f51b5ffb640 1 -- 172.21.15.28:0/2245168179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f51b00e7b30 con 0x7f51b81a6450 2024-09-21T05:48:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.662+0000 7f519effd640 1 -- 172.21.15.28:0/814263303 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac03f070 con 0x563a21ae71b0 2024-09-21T05:48:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.662+0000 7f519effd640 1 -- 172.21.15.28:0/814263303 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ac0316a0 con 0x563a21ae71b0 2024-09-21T05:48:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.662+0000 7f519effd640 1 -- 172.21.15.28:0/814263303 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac035070 con 0x563a21ae71b0 2024-09-21T05:48:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.662+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 0x7f51a00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 -- 172.21.15.28:0/814263303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 msgr2=0x563a21eb7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f51ac000c00 tx=0x7f51ac032ce0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 -- 172.21.15.28:0/814263303 shutdown_connections 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21aef390 0x563a21eb7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21eb7800 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 --2- 172.21.15.28:0/814263303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21b10070 0x563a21eb72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 0x7f51a00b8af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f51a80058c0 tx=0x7f51a800c6f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.663+0000 7f51c54c6740 1 -- 172.21.15.28:0/814263303 >> 172.21.15.28:0/814263303 conn(0x563a21b04460 msgr2=0x563a21eb51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.664+0000 7f51c54c6740 1 -- 172.21.15.28:0/814263303 shutdown_connections 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.664+0000 7f51c54c6740 1 -- 172.21.15.28:0/814263303 wait complete. 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.664+0000 7f51c54c6740 1 -- start start 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.664+0000 7f51c54c6740 1 Processor -- start 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.665+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.665+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21ec2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.665+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 0x563a21eced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.665+0000 7f51c54c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a21c7e700 con 0x563a21b10070 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51b7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21ec2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bccee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51c54c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563a21db8160 con 0x563a21ae71b0 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51c54c6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563a21ec5a40 con 0x563a21aef390 2024-09-21T05:48:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bccee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53822/0 (socket says 172.21.15.28:53822) 2024-09-21T05:48:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bccee640 1 -- 172.21.15.28:0/1271391454 learned_addr learned my addr 172.21.15.28:0/1271391454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 0x563a21eced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51c54c6740 10 client.0 _ll_register_callbacks cb 0x563a21dd19e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:06.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 -- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 msgr2=0x563a21ec2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21ec2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 -- 172.21.15.28:0/1271391454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 msgr2=0x563a21ec2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bd4ef640 1 -- 172.21.15.28:0/1271391454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a21de5c90 con 0x563a21b10070 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.666+0000 7f51bccee640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 0x563a21eced60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f51b00e52a0 tx=0x7f51b00ad3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16ae68afce2885b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f5197fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00ad4b0 con 0x563a21b10070 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f51b000fc50 con 0x563a21b10070 2024-09-21T05:48:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51ac0316a0 con 0x563a21b10070 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f51b00ae0a0 con 0x563a21b10070 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f51c54c6740 10 client.16170 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.667+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a21ed1700 con 0x563a21b10070 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0097020 con 0x563a21b10070 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f51b00b0ce0 con 0x563a21b10070 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 10 client.16170 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 20 client.16170 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21ace9f0 0x563a21ed4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 4 client.16170 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f51ac0316a0 con 0x563a21ace9f0 2024-09-21T05:48:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.668+0000 7f51bccee640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21ace9f0 0x563a21ed4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.669+0000 7f51bccee640 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21ace9f0 0x563a21ed4c70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84fd4d630fd45b93 server_cookie=9a9ec7ed7236bd4c in_seq=0 out_seq=0 2024-09-21T05:48:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.669+0000 7f5197fff640 10 client.16170 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.670+0000 7f5197fff640 1 -- 172.21.15.28:0/1271391454 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f51ac0316a0 con 0x563a21ace9f0 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.670+0000 7f5197fff640 10 client.16170 handle_command_reply: tid=0 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.670+0000 7f51c54c6740 1 client.16170 shutdown 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 msgr2=0x563a21eced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 0x563a21eced60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f51b00e52a0 tx=0x7f51b00ad3c0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 shutdown_connections 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21ace9f0 0x563a21ed4c70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f5197fff640 0 client.16170 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21b10070 0x563a21eced60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21ec2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 --2- 172.21.15.28:0/1271391454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ae71b0 0x563a21ec2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 >> 172.21.15.28:0/1271391454 conn(0x563a21b04460 msgr2=0x563a21eb8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 shutdown_connections 2024-09-21T05:48:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 1 -- 172.21.15.28:0/1271391454 wait complete. 2024-09-21T05:48:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.671+0000 7f51c54c6740 20 client.16170 trim_cache size 0 max 16384 2024-09-21T05:48:06.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.685+0000 7f51c54c6740 1 Processor -- start 2024-09-21T05:48:06.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 -- start start 2024-09-21T05:48:06.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 0x563a21ecec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 0x563a21ebfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b5f10 con 0x563a21ace9f0 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b7a50 con 0x563a21aef390 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51c54c6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b65e0 con 0x563a21ae71b0 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51bccee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51bccee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54038/0 (socket says 172.21.15.28:54038) 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51bccee640 1 -- 172.21.15.28:0/3082141274 learned_addr learned my addr 172.21.15.28:0/3082141274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51b7fff640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 0x563a21ecec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 0x563a21ebfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:06 smithi028 ceph-mon[31227]: pgmap v357: 97 pgs: 97 active+clean; 3.4 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 500 KiB/s wr, 176 op/s 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.686+0000 7f51bccee640 1 -- 172.21.15.28:0/3082141274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 msgr2=0x563a21ecec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bccee640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 0x563a21ecec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bccee640 1 -- 172.21.15.28:0/3082141274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 msgr2=0x563a21ebfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bccee640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 0x563a21ebfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bccee640 1 -- 172.21.15.28:0/3082141274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a21ec3be0 con 0x563a21ace9f0 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51b7fff640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 0x563a21ecec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 0x563a21ebfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f51bccee640 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f51ac009900 tx=0x7f51ac02aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd534929d8877a20 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f519e7fc640 1 -- 172.21.15.28:0/3082141274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac035030 con 0x563a21ace9f0 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f519e7fc640 1 -- 172.21.15.28:0/3082141274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ac0318a0 con 0x563a21ace9f0 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.687+0000 7f519e7fc640 1 -- 172.21.15.28:0/3082141274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac03f040 con 0x563a21ace9f0 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 -- 172.21.15.28:0/3082141274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 msgr2=0x563a21eb8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f51ac009900 tx=0x7f51ac02aa90 comp rx=0 tx=0).stop 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 -- 172.21.15.28:0/3082141274 shutdown_connections 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21aef390 0x563a21ebfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21ae71b0 0x563a21ecec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 --2- 172.21.15.28:0/3082141274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ace9f0 0x563a21eb8a00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.688+0000 7f51c54c6740 1 -- 172.21.15.28:0/3082141274 >> 172.21.15.28:0/3082141274 conn(0x563a21b04460 msgr2=0x563a21eb82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.689+0000 7f51c54c6740 1 -- 172.21.15.28:0/3082141274 shutdown_connections 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.689+0000 7f51c54c6740 1 -- 172.21.15.28:0/3082141274 wait complete. 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.689+0000 7f51c54c6740 1 -- start start 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.689+0000 7f51c54c6740 1 Processor -- start 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 0x563a21ec6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21f3e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b7380 con 0x563a21ae71b0 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b6cb0 con 0x563a21ace9f0 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81b3ef0 con 0x563a21aef390 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54044/0 (socket says 172.21.15.28:54044) 2024-09-21T05:48:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51b7fff640 1 -- 172.21.15.28:0/2795368810 learned_addr learned my addr 172.21.15.28:0/2795368810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 0x563a21ec6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51c54c6740 10 client.0 _ll_register_callbacks cb 0x7f51b0018400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21f3e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 -- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 msgr2=0x563a21f3e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21f3e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 -- 172.21.15.28:0/2795368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 msgr2=0x563a21ec7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.690+0000 7f51bccee640 1 -- 172.21.15.28:0/2795368810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a21ec7c80 con 0x563a21ace9f0 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51bd4ef640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21f3e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51bccee640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 0x563a21ec6f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51ac005f70 tx=0x7f51ac000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f86900378b5ce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f5197fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac035070 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f51ac003ac0 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51ac0318a0 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f51ac03f040 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51c54c6740 10 client.25763 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.691+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a21f3fb20 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ac0318a0 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f51ac057710 con 0x563a21ace9f0 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 10 client.25763 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 20 client.25763 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21b10070 0x563a21f41e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 4 client.25763 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a21e2fdf0 con 0x563a21b10070 2024-09-21T05:48:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.692+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21b10070 0x563a21f41e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f5197fff640 10 client.25763 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51b7fff640 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21b10070 0x563a21f41e50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f3b514901bf9471 server_cookie=41095d0997520a8e in_seq=0 out_seq=0 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f5197fff640 1 -- 172.21.15.28:0/2795368810 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a21e2fdf0 con 0x563a21b10070 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f5197fff640 10 client.25763 handle_command_reply: tid=0 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 client.25763 shutdown 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 msgr2=0x563a21ec6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 0x563a21ec6f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51ac005f70 tx=0x7f51ac000bc0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 shutdown_connections 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563a21b10070 0x563a21f41e50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f5197fff640 0 client.25763 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563a21aef390 0x563a21f3e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a21ae71b0 0x563a21ec7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 --2- 172.21.15.28:0/2795368810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a21ace9f0 0x563a21ec6f50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 >> 172.21.15.28:0/2795368810 conn(0x563a21b04460 msgr2=0x563a21ec4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 shutdown_connections 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.693+0000 7f51c54c6740 1 -- 172.21.15.28:0/2795368810 wait complete. 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51c54c6740 20 client.25763 trim_cache size 0 max 16384 2024-09-21T05:48:06.702 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 msgr2=0x7f51a00b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 0x7f51a00b8af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f51a80058c0 tx=0x7f51a800c6f0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 msgr2=0x7f51b81a6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f51b000cce0 tx=0x7f51b000a620 comp rx=0 tx=0).stop 2024-09-21T05:48:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 shutdown_connections 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51a00b65e0 0x7f51a00b8af0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51b81a6450 0x7f51b81a6860 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51b810de10 0x7f51b810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 --2- 172.21.15.28:0/2245168179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51b8105300 0x7f51b810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.694+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 >> 172.21.15.28:0/2245168179 conn(0x7f51b8079730 msgr2=0x7f51b81129d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.695+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 shutdown_connections 2024-09-21T05:48:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:06.695+0000 7f51bef52640 1 -- 172.21.15.28:0/2245168179 wait complete. 2024-09-21T05:48:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:06 smithi181 ceph-mon[34819]: pgmap v357: 97 pgs: 97 active+clean; 3.4 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 500 KiB/s wr, 176 op/s 2024-09-21T05:48:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:06 smithi129 ceph-mon[34885]: pgmap v357: 97 pgs: 97 active+clean; 3.4 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 500 KiB/s wr, 176 op/s 2024-09-21T05:48:08.392 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:48:08.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:48:08.746 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:08 smithi181 ceph-mon[34819]: pgmap v358: 97 pgs: 97 active+clean; 844 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 71 MiB/s wr, 218 op/s 2024-09-21T05:48:08.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:08 smithi028 ceph-mon[31227]: pgmap v358: 97 pgs: 97 active+clean; 844 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 71 MiB/s wr, 218 op/s 2024-09-21T05:48:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:08 smithi129 ceph-mon[34885]: pgmap v358: 97 pgs: 97 active+clean; 844 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 71 MiB/s wr, 218 op/s 2024-09-21T05:48:10.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:09 smithi181 ceph-mon[34819]: pgmap v359: 97 pgs: 97 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 120 MiB/s wr, 254 op/s 2024-09-21T05:48:10.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:09 smithi028 ceph-mon[31227]: pgmap v359: 97 pgs: 97 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 120 MiB/s wr, 254 op/s 2024-09-21T05:48:10.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:09 smithi129 ceph-mon[34885]: pgmap v359: 97 pgs: 97 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 120 MiB/s wr, 254 op/s 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/4189015791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4111ca0 msgr2=0x7fa4a41140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/4189015791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4111ca0 0x7fa4a41140c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa494009900 tx=0x7fa49402a3f0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/4189015791 shutdown_connections 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/4189015791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4a4114600 0x7fa4a4116a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/4189015791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4111ca0 0x7fa4a41140c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/4189015791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a4102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/4189015791 >> 172.21.15.28:0/4189015791 conn(0x7fa4a4077cc0 msgr2=0x7fa4a4078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/4189015791 shutdown_connections 2024-09-21T05:48:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.650+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/4189015791 wait complete. 2024-09-21T05:48:10.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.651+0000 7fa4a9cbc640 1 Processor -- start 2024-09-21T05:48:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.652+0000 7fa4a9cbc640 1 -- start start 2024-09-21T05:48:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.652+0000 7fa4a9cbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a9cbc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4a4111ca0 0x7fa4a41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a9cbc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4114600 0x7fa4a41a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a9cbc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a411a110 con 0x7fa4a4101ff0 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a9cbc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a4119f90 con 0x7fa4a4114600 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a9cbc640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a411a290 con 0x7fa4a4111ca0 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54070/0 (socket says 172.21.15.28:54070) 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.653+0000 7fa4a37fe640 1 -- 172.21.15.28:0/2420167756 learned_addr learned my addr 172.21.15.28:0/2420167756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 -- 172.21.15.28:0/2420167756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4a4111ca0 msgr2=0x7fa4a41a1180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:48:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4a4111ca0 0x7fa4a41a1180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 -- 172.21.15.28:0/2420167756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4114600 msgr2=0x7fa4a41a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4114600 0x7fa4a41a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 -- 172.21.15.28:0/2420167756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a41a6b50 con 0x7fa4a4101ff0 2024-09-21T05:48:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.654+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e9c0 tx=0x7fa49800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.655+0000 7fa4a0ff9640 1 -- 172.21.15.28:0/2420167756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49800cd80 con 0x7fa4a4101ff0 2024-09-21T05:48:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.655+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a41a6e40 con 0x7fa4a4101ff0 2024-09-21T05:48:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.655+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a41ad5a0 con 0x7fa4a4101ff0 2024-09-21T05:48:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.655+0000 7fa4a0ff9640 1 -- 172.21.15.28:0/2420167756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4980035f0 con 0x7fa4a4101ff0 2024-09-21T05:48:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.655+0000 7fa4a0ff9640 1 -- 172.21.15.28:0/2420167756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498010690 con 0x7fa4a4101ff0 2024-09-21T05:48:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.660+0000 7fa4b0230740 1 Processor -- start 2024-09-21T05:48:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.660+0000 7fa4b0230740 1 -- start start 2024-09-21T05:48:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.661+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 0x55cc8c509430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.661+0000 7fa4a0ff9640 1 -- 172.21.15.28:0/2420167756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa4980108f0 con 0x7fa4a4101ff0 2024-09-21T05:48:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 0x55cc8c509970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c509eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4b0230740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c47fe60 con 0x55cc8c139490 2024-09-21T05:48:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4b0230740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c46f290 con 0x55cc8c141670 2024-09-21T05:48:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 0x55cc8c509430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.662+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 0x55cc8c509430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:45508/0 (socket says 172.21.15.28:45508) 2024-09-21T05:48:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.663+0000 7fa4a3fff640 1 -- 172.21.15.28:0/3955860273 learned_addr learned my addr 172.21.15.28:0/3955860273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.663+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c4730a0 con 0x55cc8c162350 2024-09-21T05:48:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.663+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 0x55cc8c509970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a37fe640 1 -- 172.21.15.28:0/3955860273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 msgr2=0x55cc8c509430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 0x55cc8c509430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c509eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a37fe640 1 -- 172.21.15.28:0/3955860273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 msgr2=0x55cc8c509eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c509eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a37fe640 1 -- 172.21.15.28:0/3955860273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc8c1cd930 con 0x55cc8c139490 2024-09-21T05:48:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.664+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c509eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa4a0ff9640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 0x7fa48c0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 0x55cc8c509970 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa4980e51a0 tx=0x7fa4980ebe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa11abc7b40579ce server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 0x7fa48c0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa489ffb640 1 -- 172.21.15.28:0/3955860273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4980a9ef0 con 0x55cc8c139490 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa489ffb640 1 -- 172.21.15.28:0/3955860273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4980ee070 con 0x55cc8c139490 2024-09-21T05:48:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.665+0000 7fa489ffb640 1 -- 172.21.15.28:0/3955860273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4980ad010 con 0x55cc8c139490 2024-09-21T05:48:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.667+0000 7fa4a0ff9640 1 -- 172.21.15.28:0/2420167756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa4980e7a30 con 0x7fa4a4101ff0 2024-09-21T05:48:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.667+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 0x7fa48c0b8c40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0092f0 tx=0x7fa49c009600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.667+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 msgr2=0x55cc8c509970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.667+0000 7fa4b0230740 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 0x55cc8c509970 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa4980e51a0 tx=0x7fa4980ebe80 comp rx=0 tx=0).stop 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 shutdown_connections 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c509eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c139490 0x55cc8c509970 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 --2- 172.21.15.28:0/3955860273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c162350 0x55cc8c509430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 >> 172.21.15.28:0/3955860273 conn(0x55cc8c156740 msgr2=0x55cc8c507320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 shutdown_connections 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.668+0000 7fa4b0230740 1 -- 172.21.15.28:0/3955860273 wait complete. 2024-09-21T05:48:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.669+0000 7fa4b0230740 1 -- start start 2024-09-21T05:48:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.669+0000 7fa4b0230740 1 Processor -- start 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c5150d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c515690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c2d0890 con 0x55cc8c162350 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c409d40 con 0x55cc8c141670 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54090/0 (socket says 172.21.15.28:54090) 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4a3fff640 1 -- 172.21.15.28:0/2349034717 learned_addr learned my addr 172.21.15.28:0/2349034717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cc8c517c10 con 0x55cc8c139490 2024-09-21T05:48:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4b0230740 10 client.0 _ll_register_callbacks cb 0x55cc8c46dc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.670+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c5150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 -- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 msgr2=0x55cc8c5150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c5150d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 -- 172.21.15.28:0/2349034717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 msgr2=0x55cc8c515690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c515690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 -- 172.21.15.28:0/2349034717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc8c5215d0 con 0x55cc8c162350 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa49c012820 tx=0x7fa49c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0fae1d38a38591 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c01da40 con 0x55cc8c162350 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa49c01dbe0 con 0x55cc8c162350 2024-09-21T05:48:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.671+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4980ee070 con 0x55cc8c162350 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.672+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa49c01dde0 con 0x55cc8c162350 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.672+0000 7fa4b0230740 10 client.16188 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.672+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc8c523eb0 con 0x55cc8c162350 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.672+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4980ee070 con 0x55cc8c162350 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa49c00f040 con 0x55cc8c162350 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 10 client.16188 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 20 client.16188 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c120cd0 0x55cc8c527430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 4 client.16188 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc8c481db0 con 0x55cc8c120cd0 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.673+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c120cd0 0x55cc8c527430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.674+0000 7fa482ffd640 10 client.16188 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.674+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c120cd0 0x55cc8c527430 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=276125439826fbb4 server_cookie=afd071b8fc754c02 in_seq=0 out_seq=0 2024-09-21T05:48:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.675+0000 7fa482ffd640 1 -- 172.21.15.28:0/2349034717 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cc8c481db0 con 0x55cc8c120cd0 2024-09-21T05:48:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.675+0000 7fa482ffd640 10 client.16188 handle_command_reply: tid=0 2024-09-21T05:48:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 client.16188 shutdown 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 msgr2=0x55cc8c520f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa49c012820 tx=0x7fa49c004a40 comp rx=0 tx=0).stop 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 shutdown_connections 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c120cd0 0x55cc8c527430 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.676+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c162350 0x55cc8c520f60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c141670 0x55cc8c515690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa482ffd640 0 client.16188 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 1 --2- 172.21.15.28:0/2349034717 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c5150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 >> 172.21.15.28:0/2349034717 conn(0x55cc8c156740 msgr2=0x55cc8c50a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 shutdown_connections 2024-09-21T05:48:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 1 -- 172.21.15.28:0/2349034717 wait complete. 2024-09-21T05:48:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.677+0000 7fa4b0230740 20 client.16188 trim_cache size 0 max 16384 2024-09-21T05:48:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.716+0000 7fa4b0230740 1 Processor -- start 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.716+0000 7fa4b0230740 1 -- start start 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 0x55cc8c50a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c50a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c50e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 0x55cc8c50a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 0x55cc8c50a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53894/0 (socket says 172.21.15.28:53894) 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4a37fe640 1 -- 172.21.15.28:0/1829707326 learned_addr learned my addr 172.21.15.28:0/1829707326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.717+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c50a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.718+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa49000e7b0 con 0x55cc8c141670 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.718+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900102f0 con 0x55cc8c120cd0 2024-09-21T05:48:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.718+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa49000ee80 con 0x55cc8c139490 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.718+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c50e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.720+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/1829707326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 msgr2=0x55cc8c50a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.720+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 0x55cc8c50a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.720+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/1829707326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 msgr2=0x55cc8c50e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.720+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c50e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.720+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/1829707326 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc8c507860 con 0x55cc8c139490 2024-09-21T05:48:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.723+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c50a9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4a41127f0 tx=0x7fa494005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abf81fae2df1efe2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.725+0000 7fa4897fa640 1 -- 172.21.15.28:0/1829707326 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49403f070 con 0x55cc8c139490 2024-09-21T05:48:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.725+0000 7fa4897fa640 1 -- 172.21.15.28:0/1829707326 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa494004410 con 0x55cc8c139490 2024-09-21T05:48:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.725+0000 7fa4897fa640 1 -- 172.21.15.28:0/1829707326 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494035030 con 0x55cc8c139490 2024-09-21T05:48:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 msgr2=0x55cc8c50a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c50a9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4a41127f0 tx=0x7fa494005f30 comp rx=0 tx=0).stop 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 shutdown_connections 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c50e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c139490 0x55cc8c50a9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 --2- 172.21.15.28:0/1829707326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c120cd0 0x55cc8c50a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.727+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 >> 172.21.15.28:0/1829707326 conn(0x55cc8c156740 msgr2=0x55cc8c513750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.730+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 shutdown_connections 2024-09-21T05:48:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.731+0000 7fa4b0230740 1 -- 172.21.15.28:0/1829707326 wait complete. 2024-09-21T05:48:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.732+0000 7fa4b0230740 1 -- start start 2024-09-21T05:48:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.733+0000 7fa4b0230740 1 Processor -- start 2024-09-21T05:48:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.734+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c120cd0 0x55cc8c519a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.734+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.735+0000 7fa4b0230740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c58af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.735+0000 7fa4b0230740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa49000fc20 con 0x55cc8c141670 2024-09-21T05:48:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.735+0000 7fa4b0230740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa49000f550 con 0x55cc8c139490 2024-09-21T05:48:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.735+0000 7fa4b0230740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa49000c790 con 0x55cc8c120cd0 2024-09-21T05:48:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.736+0000 7fa4b0230740 10 client.0 _ll_register_callbacks cb 0x7fa49c009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.736+0000 7fa4a37fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c120cd0 0x55cc8c519a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.736+0000 7fa4a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.736+0000 7fa4a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53904/0 (socket says 172.21.15.28:53904) 2024-09-21T05:48:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.736+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/13675580 learned_addr learned my addr 172.21.15.28:0/13675580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.737+0000 7fa4a3fff640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c58af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.741+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c120cd0 msgr2=0x55cc8c519a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.741+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c120cd0 0x55cc8c519a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.741+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/13675580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 msgr2=0x55cc8c58af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.741+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c58af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.741+0000 7fa4a2ffd640 1 -- 172.21.15.28:0/13675580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc8c58b5e0 con 0x55cc8c139490 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.742+0000 7fa4a2ffd640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa494009850 tx=0x7fa494031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a9d3899859d85c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.742+0000 7fa482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.744+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49403f030 con 0x55cc8c139490 2024-09-21T05:48:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.744+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa4940046c0 con 0x55cc8c139490 2024-09-21T05:48:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.744+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa494004410 con 0x55cc8c139490 2024-09-21T05:48:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.744+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa494056050 con 0x55cc8c139490 2024-09-21T05:48:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.745+0000 7fa4b0230740 10 client.25772 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.745+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc8c592a40 con 0x55cc8c139490 2024-09-21T05:48:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.747+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa494004410 con 0x55cc8c139490 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa494003bc0 con 0x55cc8c139490 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 10 client.25772 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 20 client.25772 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c162350 0x55cc8c595160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 4 client.25772 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:48:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.748+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cc8c47d860 con 0x55cc8c162350 2024-09-21T05:48:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.749+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c162350 0x55cc8c595160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:10.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.753+0000 7fa482ffd640 10 client.25772 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:10.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.753+0000 7fa4a37fe640 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c162350 0x55cc8c595160 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e209ea67e9d47269 server_cookie=dfffe33a314214ab in_seq=0 out_seq=0 2024-09-21T05:48:10.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa482ffd640 1 -- 172.21.15.28:0/13675580 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cc8c47d860 con 0x55cc8c162350 2024-09-21T05:48:10.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa482ffd640 10 client.25772 handle_command_reply: tid=0 2024-09-21T05:48:10.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa4b0230740 1 client.25772 shutdown 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 msgr2=0x55cc8c519fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa494009850 tx=0x7fa494031400 comp rx=0 tx=0).stop 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.755+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 shutdown_connections 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cc8c162350 0x55cc8c595160 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc8c141670 0x55cc8c58af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cc8c139490 0x55cc8c519fe0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 --2- 172.21.15.28:0/13675580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cc8c120cd0 0x55cc8c519a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 >> 172.21.15.28:0/13675580 conn(0x55cc8c156740 msgr2=0x55cc8c516380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa482ffd640 0 client.25772 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 shutdown_connections 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.756+0000 7fa4b0230740 1 -- 172.21.15.28:0/13675580 wait complete. 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.757+0000 7fa4b0230740 20 client.25772 trim_cache size 0 max 16384 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:48:10.759 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ecc4b123-4671-4fe2-b7f5-102d3296a8aa", 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 msgr2=0x7fa48c0b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 0x7fa48c0b8c40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0092f0 tx=0x7fa49c009600 comp rx=0 tx=0).stop 2024-09-21T05:48:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 msgr2=0x7fa4a41a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e9c0 tx=0x7fa49800ee80 comp rx=0 tx=0).stop 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 shutdown_connections 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4a4114600 0x7fa4a41a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4a4111ca0 0x7fa4a41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa48c0b6730 0x7fa48c0b8c40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 --2- 172.21.15.28:0/2420167756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4a4101ff0 0x7fa4a41a0be0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 >> 172.21.15.28:0/2420167756 conn(0x7fa4a4077cc0 msgr2=0x7fa4a40fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 shutdown_connections 2024-09-21T05:48:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:10.758+0000 7fa4a9cbc640 1 -- 172.21.15.28:0/2420167756 wait complete. 2024-09-21T05:48:11.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:48:11.839 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:11 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:48:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:11 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:11 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:48:12.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:11 smithi129 ceph-mon[34885]: pgmap v360: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 135 MiB/s wr, 236 op/s 2024-09-21T05:48:12.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:11 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:48:12.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:11 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:12.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:11 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:48:12.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:11 smithi028 ceph-mon[31227]: pgmap v360: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 135 MiB/s wr, 236 op/s 2024-09-21T05:48:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.278+0000 7ff4481ba640 1 -- 172.21.15.28:0/1680754908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 msgr2=0x7ff440102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.278+0000 7ff4481ba640 1 --2- 172.21.15.28:0/1680754908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff440102dc0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff438009ca0 tx=0x7ff43802a8d0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.280+0000 7ff4481ba640 1 -- 172.21.15.28:0/1680754908 shutdown_connections 2024-09-21T05:48:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.280+0000 7ff4481ba640 1 --2- 172.21.15.28:0/1680754908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff440108c30 0x7ff44010f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.280+0000 7ff4481ba640 1 --2- 172.21.15.28:0/1680754908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4401082e0 0x7ff4401086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.280+0000 7ff4481ba640 1 --2- 172.21.15.28:0/1680754908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff440102dc0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.280+0000 7ff4481ba640 1 -- 172.21.15.28:0/1680754908 >> 172.21.15.28:0/1680754908 conn(0x7ff44007a740 msgr2=0x7ff44007ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.281+0000 7ff4481ba640 1 -- 172.21.15.28:0/1680754908 shutdown_connections 2024-09-21T05:48:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.281+0000 7ff4481ba640 1 -- 172.21.15.28:0/1680754908 wait complete. 2024-09-21T05:48:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.282+0000 7ff4481ba640 1 Processor -- start 2024-09-21T05:48:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 -- start start 2024-09-21T05:48:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4401082e0 0x7ff4401a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff440108c30 0x7ff4401a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff440112a00 con 0x7ff4401029b0 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff440112880 con 0x7ff440108c30 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff4481ba640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff440112b80 con 0x7ff4401082e0 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54130/0 (socket says 172.21.15.28:54130) 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.283+0000 7ff445f56640 1 -- 172.21.15.28:0/2384986916 learned_addr learned my addr 172.21.15.28:0/2384986916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.284+0000 7ff445f56640 1 -- 172.21.15.28:0/2384986916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4401082e0 msgr2=0x7ff4401a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.284+0000 7ff445f56640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4401082e0 0x7ff4401a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.284+0000 7ff445f56640 1 -- 172.21.15.28:0/2384986916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff440108c30 msgr2=0x7ff4401a6440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.284+0000 7ff445f56640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff440108c30 0x7ff4401a6440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.284+0000 7ff445f56640 1 -- 172.21.15.28:0/2384986916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4401a6b40 con 0x7ff4401029b0 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.285+0000 7ff445f56640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff43802aec0 tx=0x7ff438003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.285+0000 7ff436ffd640 1 -- 172.21.15.28:0/2384986916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438003e20 con 0x7ff4401029b0 2024-09-21T05:48:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.286+0000 7ff436ffd640 1 -- 172.21.15.28:0/2384986916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4380044f0 con 0x7ff4401029b0 2024-09-21T05:48:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.286+0000 7ff436ffd640 1 -- 172.21.15.28:0/2384986916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43802fde0 con 0x7ff4401029b0 2024-09-21T05:48:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.286+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4401a6dd0 con 0x7ff4401029b0 2024-09-21T05:48:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.286+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4401ad5c0 con 0x7ff4401029b0 2024-09-21T05:48:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.291+0000 7ff436ffd640 1 -- 172.21.15.28:0/2384986916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff438004040 con 0x7ff4401029b0 2024-09-21T05:48:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:11 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:48:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:11 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:11 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:48:12.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:11 smithi181 ceph-mon[34819]: pgmap v360: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 135 MiB/s wr, 236 op/s 2024-09-21T05:48:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.292+0000 7ff44f207740 1 Processor -- start 2024-09-21T05:48:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.292+0000 7ff436ffd640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 0x7ff42c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.292+0000 7ff44f207740 1 -- start start 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.292+0000 7ff436ffd640 1 -- 172.21.15.28:0/2384986916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff438108bd0 con 0x7ff4401029b0 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.292+0000 7ff446757640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 0x7ff42c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6fd99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fda430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572c6f4e6f0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572c6f3db20 con 0x5572c6c341b0 2024-09-21T05:48:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff44f207740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572c6f41930 con 0x5572c6c134d0 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54134/0 (socket says 172.21.15.28:54134) 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 -- 172.21.15.28:0/680428531 learned_addr learned my addr 172.21.15.28:0/680428531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 -- 172.21.15.28:0/680428531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 msgr2=0x5572c6fda430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fda430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 -- 172.21.15.28:0/680428531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 msgr2=0x5572c6fd99b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6fd99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff445755640 1 -- 172.21.15.28:0/680428531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572c6f03530 con 0x5572c6c0b2f0 2024-09-21T05:48:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.293+0000 7ff446757640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 0x7ff42c0b8c10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff4300096c0 tx=0x7ff430006c20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.294+0000 7ff445755640 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009700 tx=0x7ff43c009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adbc0a2771a9cbd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.294+0000 7ff42bfff640 1 -- 172.21.15.28:0/680428531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c015020 con 0x5572c6c0b2f0 2024-09-21T05:48:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.294+0000 7ff42bfff640 1 -- 172.21.15.28:0/680428531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff43c009e10 con 0x5572c6c0b2f0 2024-09-21T05:48:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.294+0000 7ff42bfff640 1 -- 172.21.15.28:0/680428531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c010380 con 0x5572c6c0b2f0 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 -- 172.21.15.28:0/680428531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 msgr2=0x5572c6fd9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009700 tx=0x7ff43c009bc0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 -- 172.21.15.28:0/680428531 shutdown_connections 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fda430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fd9ef0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 --2- 172.21.15.28:0/680428531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6fd99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 -- 172.21.15.28:0/680428531 >> 172.21.15.28:0/680428531 conn(0x5572c6c28300 msgr2=0x5572c6fd78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 -- 172.21.15.28:0/680428531 shutdown_connections 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.295+0000 7ff44f207740 1 -- 172.21.15.28:0/680428531 wait complete. 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.296+0000 7ff44f207740 1 -- start start 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.296+0000 7ff44f207740 1 Processor -- start 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c134d0 0x5572c6fe5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54136/0 (socket says 172.21.15.28:54136) 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 -- 172.21.15.28:0/1377273250 learned_addr learned my addr 172.21.15.28:0/1377273250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c341b0 0x5572c6ff1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572c6e75ab0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5572c6ee3730 con 0x5572c6c134d0 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5572c6fe8130 con 0x5572c6c341b0 2024-09-21T05:48:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff44f207740 10 client.0 _ll_register_callbacks cb 0x5572c6f376b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 -- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c341b0 msgr2=0x5572c6ff1450 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c341b0 0x5572c6ff1450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 -- 172.21.15.28:0/1377273250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c134d0 msgr2=0x5572c6fe5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c134d0 0x5572c6fe5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.297+0000 7ff445f56640 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572c6c9f790 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.298+0000 7ff445f56640 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff4381063a0 tx=0x7ff4380ce270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5de2cd3233cd3d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.298+0000 7ff4297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.298+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ce550 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.298+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff4380d31d0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.298+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff43c009e10 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff4380cf140 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 10 client.16206 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5572c6d18620 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff43c009e10 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff4380cacb0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 10 client.16206 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 10 client.16206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 20 client.16206 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bf2d80 0x5572c6ff7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 4 client.16206 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.299+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572c6f50640 con 0x5572c6bf2d80 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.300+0000 7ff445755640 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bf2d80 0x5572c6ff7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.301+0000 7ff4297fa640 10 client.16206 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.301+0000 7ff445755640 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bf2d80 0x5572c6ff7220 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7d4468963a4fe3 server_cookie=b83c2391879886ba in_seq=0 out_seq=0 2024-09-21T05:48:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff4297fa640 1 -- 172.21.15.28:0/1377273250 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5572c6f50640 con 0x5572c6bf2d80 2024-09-21T05:48:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff4297fa640 10 client.16206 handle_command_reply: tid=0 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 client.16206 shutdown 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 msgr2=0x5572c6fe50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff4381063a0 tx=0x7ff4380ce270 comp rx=0 tx=0).stop 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 shutdown_connections 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bf2d80 0x5572c6ff7220 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c341b0 0x5572c6ff1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c134d0 0x5572c6fe5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff4297fa640 0 client.16206 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 --2- 172.21.15.28:0/1377273250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe50c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.304+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 >> 172.21.15.28:0/1377273250 conn(0x5572c6c28300 msgr2=0x5572c6fda9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.305+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 shutdown_connections 2024-09-21T05:48:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.305+0000 7ff44f207740 1 -- 172.21.15.28:0/1377273250 wait complete. 2024-09-21T05:48:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.305+0000 7ff44f207740 20 client.16206 trim_cache size 0 max 16384 2024-09-21T05:48:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.324+0000 7ff44f207740 1 Processor -- start 2024-09-21T05:48:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.324+0000 7ff44f207740 1 -- start start 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6ff1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b5de0 con 0x5572c6c134d0 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b7920 con 0x5572c6c341b0 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff44f207740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b64b0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54152/0 (socket says 172.21.15.28:54152) 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 -- 172.21.15.28:0/2936848110 learned_addr learned my addr 172.21.15.28:0/2936848110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 -- 172.21.15.28:0/2936848110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c0b2f0 msgr2=0x5572c6fe55a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe55a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 -- 172.21.15.28:0/2936848110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 msgr2=0x5572c6ff1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6ff1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.325+0000 7ff445755640 1 -- 172.21.15.28:0/2936848110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572c6fdf5b0 con 0x5572c6c134d0 2024-09-21T05:48:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.326+0000 7ff445755640 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x5572c6fd8630 tx=0x7ff43c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2829ebcf13ba15 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.326+0000 7ff42b7fe640 1 -- 172.21.15.28:0/2936848110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c027070 con 0x5572c6c134d0 2024-09-21T05:48:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.326+0000 7ff42b7fe640 1 -- 172.21.15.28:0/2936848110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff43c011ac0 con 0x5572c6c134d0 2024-09-21T05:48:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.326+0000 7ff42b7fe640 1 -- 172.21.15.28:0/2936848110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c015040 con 0x5572c6c134d0 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- 172.21.15.28:0/2936848110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 msgr2=0x5572c6fe6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x5572c6fd8630 tx=0x7ff43c000bc0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- 172.21.15.28:0/2936848110 shutdown_connections 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c6ff1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c134d0 0x5572c6fe6cf0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 --2- 172.21.15.28:0/2936848110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- 172.21.15.28:0/2936848110 >> 172.21.15.28:0/2936848110 conn(0x5572c6c28300 msgr2=0x5572c6fe65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- 172.21.15.28:0/2936848110 shutdown_connections 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- 172.21.15.28:0/2936848110 wait complete. 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 -- start start 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.327+0000 7ff44f207740 1 Processor -- start 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff445f56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54160/0 (socket says 172.21.15.28:54160) 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fe9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff445f56640 1 -- 172.21.15.28:0/4157910805 learned_addr learned my addr 172.21.15.28:0/4157910805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c7059c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b7250 con 0x5572c6c0b2f0 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b6b80 con 0x5572c6c341b0 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401b3dc0 con 0x5572c6c134d0 2024-09-21T05:48:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.328+0000 7ff44f207740 10 client.0 _ll_register_callbacks cb 0x7ff438004950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 -- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 msgr2=0x5572c6fe9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fe9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 -- 172.21.15.28:0/4157910805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 msgr2=0x5572c7059c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c7059c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572c705a300 con 0x5572c6c0b2f0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff445f56640 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff4380cbcb0 tx=0x7ff43802f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a4280b9ff21266 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff4297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ce390 con 0x5572c6c0b2f0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff4380b8050 con 0x5572c6c0b2f0 2024-09-21T05:48:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.329+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff43c011ac0 con 0x5572c6c0b2f0 2024-09-21T05:48:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff4380ce550 con 0x5572c6c0b2f0 2024-09-21T05:48:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 10 client.16212 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5572c7061670 con 0x5572c6c0b2f0 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4380b7020 con 0x5572c6c0b2f0 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff4380ca500 con 0x5572c6c0b2f0 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 10 client.16212 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 20 client.16212 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bc81c0 0x5572c7063d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 4 client.16212 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.330+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff43c011ac0 con 0x5572c6bc81c0 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.331+0000 7ff445755640 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bc81c0 0x5572c7063d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.332+0000 7ff4297fa640 10 client.16212 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.332+0000 7ff445755640 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bc81c0 0x5572c7063d90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51a89323de074ec4 server_cookie=f3bef272ab4678c1 in_seq=0 out_seq=0 2024-09-21T05:48:12.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 -- 172.21.15.28:0/768034921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628108cf0 msgr2=0x7f462810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 --2- 172.21.15.28:0/768034921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628108cf0 0x7f462810f520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4620009ca0 tx=0x7f462002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 -- 172.21.15.28:0/768034921 shutdown_connections 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 --2- 172.21.15.28:0/768034921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628108cf0 0x7f462810f520 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 --2- 172.21.15.28:0/768034921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46281087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 --2- 172.21.15.28:0/768034921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628102a60 0x7f4628102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 -- 172.21.15.28:0/768034921 >> 172.21.15.28:0/768034921 conn(0x7f4628079730 msgr2=0x7f4628079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 -- 172.21.15.28:0/768034921 shutdown_connections 2024-09-21T05:48:12.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.429+0000 7f462e09f640 1 -- 172.21.15.28:0/768034921 wait complete. 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.430+0000 7f462e09f640 1 Processor -- start 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.430+0000 7f462e09f640 1 -- start start 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628102a60 0x7f46280ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 0x7f46281000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4628112960 con 0x7f4628102a60 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46281127e0 con 0x7f4628108cf0 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f462e09f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4628112ae0 con 0x7f46281083a0 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 0x7f46281000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f46277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628102a60 0x7f46280ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:45606/0 (socket says 172.21.15.28:45606) 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4626ffd640 1 -- 172.21.15.28:0/2397392133 learned_addr learned my addr 172.21.15.28:0/2397392133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 -- 172.21.15.28:0/2397392133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 msgr2=0x7f46280ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 -- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628102a60 msgr2=0x7f46280ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628102a60 0x7f46280ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4627fff640 1 -- 172.21.15.28:0/2397392133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46281a8740 con 0x7f4628108cf0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.431+0000 7f4626ffd640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f4627fff640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 0x7f46281000f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f462002ade0 tx=0x7f4620031660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f4624ff9640 1 -- 172.21.15.28:0/2397392133 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620003b70 con 0x7f4628108cf0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46281a89d0 con 0x7f4628108cf0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f4624ff9640 1 -- 172.21.15.28:0/2397392133 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4620031910 con 0x7f4628108cf0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f4624ff9640 1 -- 172.21.15.28:0/2397392133 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462002f950 con 0x7f4628108cf0 2024-09-21T05:48:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.432+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46281a8f60 con 0x7f4628108cf0 2024-09-21T05:48:12.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.434+0000 7f4634613740 1 Processor -- start 2024-09-21T05:48:12.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4624ff9640 1 -- 172.21.15.28:0/2397392133 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f462002faf0 con 0x7f4628108cf0 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 -- start start 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d4c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d4d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5598f5cc2e50 con 0x5598f5984390 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598f5cb2280 con 0x5598f597c1b0 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4634613740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598f5cb6090 con 0x5598f59a5070 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d4c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53962/0 (socket says 172.21.15.28:53962) 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 -- 172.21.15.28:0/958477923 learned_addr learned my addr 172.21.15.28:0/958477923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4626ffd640 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d4d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 -- 172.21.15.28:0/958477923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 msgr2=0x5598f5d4c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d4c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 -- 172.21.15.28:0/958477923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 msgr2=0x5598f5d4d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d4d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f46277fe640 1 -- 172.21.15.28:0/958477923 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598f5a10650 con 0x5598f597c1b0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.435+0000 7f4624ff9640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 0x7f46100b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f4626ffd640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 0x7f46100b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f4624ff9640 1 -- 172.21.15.28:0/2397392133 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4620108740 con 0x7f4628108cf0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f46277fe640 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c90 tx=0x7f4618011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=970a36a062ea525e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f460dffb640 1 -- 172.21.15.28:0/958477923 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618009e40 con 0x5598f597c1b0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f460dffb640 1 -- 172.21.15.28:0/958477923 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46180054d0 con 0x5598f597c1b0 2024-09-21T05:48:12.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.436+0000 7f460dffb640 1 -- 172.21.15.28:0/958477923 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618019630 con 0x5598f597c1b0 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.437+0000 7f4634613740 1 -- 172.21.15.28:0/958477923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 msgr2=0x5598f5d4cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.437+0000 7f4626ffd640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 0x7f46100b8b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f462806a320 tx=0x7f461c004730 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.437+0000 7f4634613740 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c90 tx=0x7f4618011810 comp rx=0 tx=0).stop 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 -- 172.21.15.28:0/958477923 shutdown_connections 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d4d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d4cb90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 --2- 172.21.15.28:0/958477923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d4c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 -- 172.21.15.28:0/958477923 >> 172.21.15.28:0/958477923 conn(0x5598f5999460 msgr2=0x5598f5d4a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 -- 172.21.15.28:0/958477923 shutdown_connections 2024-09-21T05:48:12.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 -- 172.21.15.28:0/958477923 wait complete. 2024-09-21T05:48:12.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.438+0000 7f4634613740 1 -- start start 2024-09-21T05:48:12.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.439+0000 7f4634613740 1 Processor -- start 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.439+0000 7f4634613740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d60dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5598f5b13700 con 0x5598f5984390 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598f5c4d110 con 0x5598f597c1b0 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598f5d5ae30 con 0x5598f59a5070 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.440+0000 7f4634613740 10 client.0 _ll_register_callbacks cb 0x5598f5c66990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54212/0 (socket says 172.21.15.28:54212) 2024-09-21T05:48:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 -- 172.21.15.28:0/529927548 learned_addr learned my addr 172.21.15.28:0/529927548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 -- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 msgr2=0x5598f5d663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 -- 172.21.15.28:0/529927548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 msgr2=0x5598f5d60dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d60dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.441+0000 7f4626ffd640 1 -- 172.21.15.28:0/529927548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598f5c7ac40 con 0x5598f5984390 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.442+0000 7f4626ffd640 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f461c003bf0 tx=0x7f461c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6f329b730bfd62 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.442+0000 7f4606ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:12.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.442+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c030060 con 0x5598f5984390 2024-09-21T05:48:12.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.442+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46180054d0 con 0x5598f5984390 2024-09-21T05:48:12.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.442+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f461c01a050 con 0x5598f5984390 2024-09-21T05:48:12.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f461c003e60 con 0x5598f5984390 2024-09-21T05:48:12.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4634613740 10 client.16224 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598f5d6f360 con 0x5598f5984390 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46180054d0 con 0x5598f5984390 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f461c0208a0 con 0x5598f5984390 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4634613740 10 client.16224 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.443+0000 7f4634613740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.444+0000 7f4634613740 20 client.16224 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.444+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f59639f0 0x5598f5d72e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.444+0000 7f4634613740 4 client.16224 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:12.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.444+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598f5cc4da0 con 0x5598f59639f0 2024-09-21T05:48:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.450+0000 7f46277fe640 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f59639f0 0x5598f5d72e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.455+0000 7f4606ffd640 10 client.16224 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.455+0000 7f46277fe640 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f59639f0 0x5598f5d72e10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bae68e8c023ab1 server_cookie=5388e81eb45c523f in_seq=0 out_seq=0 2024-09-21T05:48:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.460+0000 7f4606ffd640 1 -- 172.21.15.28:0/529927548 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5598f5cc4da0 con 0x5598f59639f0 2024-09-21T05:48:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.460+0000 7f4606ffd640 10 client.16224 handle_command_reply: tid=0 2024-09-21T05:48:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.460+0000 7f4634613740 1 client.16224 shutdown 2024-09-21T05:48:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.461+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 msgr2=0x5598f5d613a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.461+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f461c003bf0 tx=0x7f461c0046c0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 shutdown_connections 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f59639f0 0x5598f5d72e10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d663c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d613a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 --2- 172.21.15.28:0/529927548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d60dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.464+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 >> 172.21.15.28:0/529927548 conn(0x5598f5999460 msgr2=0x5598f5d4f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.465+0000 7f4606ffd640 0 client.16224 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.466+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 shutdown_connections 2024-09-21T05:48:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.466+0000 7f4634613740 1 -- 172.21.15.28:0/529927548 wait complete. 2024-09-21T05:48:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.466+0000 7f4634613740 20 client.16224 trim_cache size 0 max 16384 2024-09-21T05:48:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff4297fa640 1 -- 172.21.15.28:0/4157910805 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff43c011ac0 con 0x5572c6bc81c0 2024-09-21T05:48:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff4297fa640 10 client.16212 handle_command_reply: tid=0 2024-09-21T05:48:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 client.16212 shutdown 2024-09-21T05:48:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 msgr2=0x5572c6fe8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff4380cbcb0 tx=0x7ff43802f850 comp rx=0 tx=0).stop 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 shutdown_connections 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5572c6bc81c0 0x5572c7063d90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5572c6c341b0 0x5572c7059c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff4297fa640 0 client.16212 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5572c6c134d0 0x5572c6fe9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 --2- 172.21.15.28:0/4157910805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572c6c0b2f0 0x5572c6fe8d80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 >> 172.21.15.28:0/4157910805 conn(0x5572c6c28300 msgr2=0x5572c6fd7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 shutdown_connections 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 1 -- 172.21.15.28:0/4157910805 wait complete. 2024-09-21T05:48:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.484+0000 7ff44f207740 20 client.16212 trim_cache size 0 max 16384 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 msgr2=0x7ff42c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 0x7ff42c0b8c10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff4300096c0 tx=0x7ff430006c20 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 msgr2=0x7ff4401a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff43802aec0 tx=0x7ff438003970 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 shutdown_connections 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff440108c30 0x7ff4401a6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4401082e0 0x7ff4401a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff42c0b6700 0x7ff42c0b8c10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 --2- 172.21.15.28:0/2384986916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4401029b0 0x7ff4401a0c10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 >> 172.21.15.28:0/2384986916 conn(0x7ff44007a740 msgr2=0x7ff440106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.485+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 shutdown_connections 2024-09-21T05:48:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:12.486+0000 7ff4481ba640 1 -- 172.21.15.28:0/2384986916 wait complete. 2024-09-21T05:48:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.491+0000 7f4634613740 1 Processor -- start 2024-09-21T05:48:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.491+0000 7f4634613740 1 -- start start 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f5984390 0x5598f5d5c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d5f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46281ad5a0 con 0x5598f597c1b0 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46281af0e0 con 0x5598f5984390 2024-09-21T05:48:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4634613740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46281adc70 con 0x5598f59a5070 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f46277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f46277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54226/0 (socket says 172.21.15.28:54226) 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f46277fe640 1 -- 172.21.15.28:0/2293704577 learned_addr learned my addr 172.21.15.28:0/2293704577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4627fff640 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d5f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.492+0000 7f4626ffd640 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f5984390 0x5598f5d5c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.493+0000 7f46277fe640 1 -- 172.21.15.28:0/2293704577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 msgr2=0x5598f5d5f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.493+0000 7f46277fe640 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d5f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.493+0000 7f46277fe640 1 -- 172.21.15.28:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f5984390 msgr2=0x5598f5d5c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.493+0000 7f46277fe640 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f5984390 0x5598f5d5c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.493+0000 7f46277fe640 1 -- 172.21.15.28:0/2293704577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598f5d667e0 con 0x5598f597c1b0 2024-09-21T05:48:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.495+0000 7f46277fe640 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f46180058c0 tx=0x7f4618004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fa3ac9326b256d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.496+0000 7f460d7fa640 1 -- 172.21.15.28:0/2293704577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461802f020 con 0x5598f597c1b0 2024-09-21T05:48:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.496+0000 7f460d7fa640 1 -- 172.21.15.28:0/2293704577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46180035f0 con 0x5598f597c1b0 2024-09-21T05:48:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.496+0000 7f460d7fa640 1 -- 172.21.15.28:0/2293704577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618019650 con 0x5598f597c1b0 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 -- 172.21.15.28:0/2293704577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 msgr2=0x5598f5d5bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f46180058c0 tx=0x7f4618004c50 comp rx=0 tx=0).stop 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 -- 172.21.15.28:0/2293704577 shutdown_connections 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5d5f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f5984390 0x5598f5d5c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 --2- 172.21.15.28:0/2293704577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f597c1b0 0x5598f5d5bd70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 -- 172.21.15.28:0/2293704577 >> 172.21.15.28:0/2293704577 conn(0x5598f5999460 msgr2=0x5598f5d45280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 -- 172.21.15.28:0/2293704577 shutdown_connections 2024-09-21T05:48:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.498+0000 7f4634613740 1 -- 172.21.15.28:0/2293704577 wait complete. 2024-09-21T05:48:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.499+0000 7f4634613740 1 -- start start 2024-09-21T05:48:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.499+0000 7f4634613740 1 Processor -- start 2024-09-21T05:48:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.500+0000 7f4634613740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d69c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4634613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d6a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d6a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f46277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d69c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f46277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d69c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:53990/0 (socket says 172.21.15.28:53990) 2024-09-21T05:48:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f46277fe640 1 -- 172.21.15.28:0/3559435752 learned_addr learned my addr 172.21.15.28:0/3559435752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5ddb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46281aea10 con 0x5598f5984390 2024-09-21T05:48:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46281ae340 con 0x5598f597c1b0 2024-09-21T05:48:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46281ab5a0 con 0x5598f59a5070 2024-09-21T05:48:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.501+0000 7f4627fff640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5ddb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4634613740 10 client.0 _ll_register_callbacks cb 0x7f461c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4626ffd640 1 -- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 msgr2=0x5598f5ddb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4626ffd640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5ddb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4626ffd640 1 -- 172.21.15.28:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 msgr2=0x5598f5d69c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4626ffd640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d69c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.502+0000 7f4626ffd640 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598f5ddbd50 con 0x5598f5984390 2024-09-21T05:48:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4626ffd640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d6a1b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f461c009f50 tx=0x7f461c01ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a024c1029966c3ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4606ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c030040 con 0x5598f5984390 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f461c028040 con 0x5598f5984390 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46180035f0 con 0x5598f5984390 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.503+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f461c045050 con 0x5598f5984390 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.504+0000 7f4634613740 10 client.16230 fetch_fsmap learned FSMap version 27 2024-09-21T05:48:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.504+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598f5ddce30 con 0x5598f5984390 2024-09-21T05:48:12.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f461c01f4a0 con 0x5598f5984390 2024-09-21T05:48:12.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f461c03c070 con 0x5598f5984390 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 10 client.16230 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 20 client.16230 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f5938da0 0x5598f5ddf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 4 client.16230 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:48:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.505+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f46180035f0 con 0x5598f5938da0 2024-09-21T05:48:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.508+0000 7f46277fe640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f5938da0 0x5598f5ddf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.513+0000 7f46277fe640 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f5938da0 0x5598f5ddf620 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76143ae2fbfceffb server_cookie=33b921b6afc4da9b in_seq=0 out_seq=0 2024-09-21T05:48:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.513+0000 7f4606ffd640 10 client.16230 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4606ffd640 1 -- 172.21.15.28:0/3559435752 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1706 (crc 0 0 0) 0x7f46180035f0 con 0x5598f5938da0 2024-09-21T05:48:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4606ffd640 10 client.16230 handle_command_reply: tid=0 2024-09-21T05:48:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4634613740 1 client.16230 shutdown 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 msgr2=0x5598f5d6a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d6a1b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f461c009f50 tx=0x7f461c01ff10 comp rx=0 tx=0).stop 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 shutdown_connections 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.514+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5598f5938da0 0x5598f5ddf620 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598f59a5070 0x5598f5ddb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4606ffd640 0 client.16230 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5598f5984390 0x5598f5d6a1b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 --2- 172.21.15.28:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598f597c1b0 0x5598f5d69c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 >> 172.21.15.28:0/3559435752 conn(0x5598f5999460 msgr2=0x5598f5d5bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 shutdown_connections 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 1 -- 172.21.15.28:0/3559435752 wait complete. 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.515+0000 7f4634613740 20 client.16230 trim_cache size 0 max 16384 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:48:12.518 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ecc4b123-4671-4fe2-b7f5-102d3296a8aa", 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:12.520 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.516+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 msgr2=0x7f46100b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.516+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 0x7f46100b8b90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f462806a320 tx=0x7f461c004730 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.516+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 msgr2=0x7f46281000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.516+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 0x7f46281000f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f462002ade0 tx=0x7f4620031660 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 shutdown_connections 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4628108cf0 0x7f46281000f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46281083a0 0x7f46280ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f46100b6680 0x7f46100b8b90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 --2- 172.21.15.28:0/2397392133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4628102a60 0x7f46280ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 >> 172.21.15.28:0/2397392133 conn(0x7f4628079730 msgr2=0x7f462806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 shutdown_connections 2024-09-21T05:48:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:12.517+0000 7f462e09f640 1 -- 172.21.15.28:0/2397392133 wait complete. 2024-09-21T05:48:13.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:12 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:13.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:12 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:13.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:12 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:48:13.424 INFO:tasks.cephfs.filesystem:scrub status for tag:ecc4b123-4671-4fe2-b7f5-102d3296a8aa - {'path': '/', 'tag': 'ecc4b123-4671-4fe2-b7f5-102d3296a8aa', 'options': 'recursive,force'} 2024-09-21T05:48:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:13 smithi181 ceph-mon[34819]: pgmap v361: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 325 MiB/s wr, 325 op/s 2024-09-21T05:48:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:13 smithi028 ceph-mon[31227]: pgmap v361: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 325 MiB/s wr, 325 op/s 2024-09-21T05:48:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:13 smithi129 ceph-mon[34885]: pgmap v361: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 325 MiB/s wr, 325 op/s 2024-09-21T05:48:16.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:15 smithi181 ceph-mon[34819]: pgmap v362: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 373 MiB/s wr, 268 op/s 2024-09-21T05:48:16.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:15 smithi028 ceph-mon[31227]: pgmap v362: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 373 MiB/s wr, 268 op/s 2024-09-21T05:48:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:15 smithi129 ceph-mon[34885]: pgmap v362: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 373 MiB/s wr, 268 op/s 2024-09-21T05:48:17.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:16 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:48:17.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:16 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:48:17.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:16 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:48:18.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:17 smithi181 ceph-mon[34819]: pgmap v363: 97 pgs: 97 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 541 MiB/s wr, 350 op/s 2024-09-21T05:48:18.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:17 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:17 smithi028 ceph-mon[31227]: pgmap v363: 97 pgs: 97 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 541 MiB/s wr, 350 op/s 2024-09-21T05:48:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:17 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:18.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:17 smithi129 ceph-mon[34885]: pgmap v363: 97 pgs: 97 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 541 MiB/s wr, 350 op/s 2024-09-21T05:48:18.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:17 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:20.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:20 smithi181 ceph-mon[34819]: pgmap v364: 97 pgs: 97 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 542 MiB/s wr, 330 op/s 2024-09-21T05:48:20.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:19 smithi028 ceph-mon[31227]: pgmap v364: 97 pgs: 97 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 542 MiB/s wr, 330 op/s 2024-09-21T05:48:20.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:20 smithi129 ceph-mon[34885]: pgmap v364: 97 pgs: 97 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 542 MiB/s wr, 330 op/s 2024-09-21T05:48:20.395 INFO:tasks.workunit.client.0.smithi028.stdout:fs setup took 17 secs 2024-09-21T05:48:22.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:22 smithi028 ceph-mon[31227]: pgmap v365: 97 pgs: 97 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 511 MiB/s wr, 276 op/s 2024-09-21T05:48:22.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:22 smithi129 ceph-mon[34885]: pgmap v365: 97 pgs: 97 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 511 MiB/s wr, 276 op/s 2024-09-21T05:48:22.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:22 smithi181 ceph-mon[34819]: pgmap v365: 97 pgs: 97 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 511 MiB/s wr, 276 op/s 2024-09-21T05:48:24.063 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...3 sec 2024-09-21T05:48:24.064 INFO:tasks.workunit.client.0.smithi028.stdout:Starting Actual Benchmark At: Sat Sep 21 05:48:24 2024 2024-09-21T05:48:24.064 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:48:24.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:24 smithi028 ceph-mon[31227]: pgmap v366: 97 pgs: 97 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 722 MiB/s wr, 356 op/s 2024-09-21T05:48:24.346 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:24 smithi181 ceph-mon[34819]: pgmap v366: 97 pgs: 97 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 722 MiB/s wr, 356 op/s 2024-09-21T05:48:24.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:24 smithi129 ceph-mon[34885]: pgmap v366: 97 pgs: 97 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 722 MiB/s wr, 356 op/s 2024-09-21T05:48:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 -- 172.21.15.28:0/1688548113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109400 msgr2=0x7fbb7c109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 --2- 172.21.15.28:0/1688548113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109400 0x7fbb7c109810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009900 tx=0x7fbb6c02a3f0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 -- 172.21.15.28:0/1688548113 shutdown_connections 2024-09-21T05:48:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 --2- 172.21.15.28:0/1688548113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c109d50 0x7fbb7c110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 --2- 172.21.15.28:0/1688548113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109400 0x7fbb7c109810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 --2- 172.21.15.28:0/1688548113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c103ad0 0x7fbb7c103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 -- 172.21.15.28:0/1688548113 >> 172.21.15.28:0/1688548113 conn(0x7fbb7c0ff510 msgr2=0x7fbb7c101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.672+0000 7fbb82030640 1 -- 172.21.15.28:0/1688548113 shutdown_connections 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.673+0000 7fbb82030640 1 -- 172.21.15.28:0/1688548113 wait complete. 2024-09-21T05:48:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.674+0000 7fbb82030640 1 Processor -- start 2024-09-21T05:48:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.674+0000 7fbb82030640 1 -- start start 2024-09-21T05:48:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.674+0000 7fbb82030640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c103ad0 0x7fbb7c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.674+0000 7fbb82030640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 0x7fbb7c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.674+0000 7fbb82030640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 0x7fbb7c19ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb82030640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c113a60 con 0x7fbb7c103ad0 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb82030640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1138e0 con 0x7fbb7c109400 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb82030640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c113be0 con 0x7fbb7c109d50 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 0x7fbb7c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 0x7fbb7c199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59434/0 (socket says 172.21.15.28:59434) 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb7affd640 1 -- 172.21.15.28:0/719421776 learned_addr learned my addr 172.21.15.28:0/719421776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.675+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 0x7fbb7c19ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c103ad0 0x7fbb7c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 -- 172.21.15.28:0/719421776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 msgr2=0x7fbb7c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 0x7fbb7c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 -- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c103ad0 msgr2=0x7fbb7c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c103ad0 0x7fbb7c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 -- 172.21.15.28:0/719421776 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c1a0470 con 0x7fbb7c109d50 2024-09-21T05:48:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.676+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 0x7fbb7c19ff30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400ec50 tx=0x7fbb7400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.677+0000 7fbb78ff9640 1 -- 172.21.15.28:0/719421776 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74003b70 con 0x7fbb7c109d50 2024-09-21T05:48:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.677+0000 7fbb78ff9640 1 -- 172.21.15.28:0/719421776 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb74003d10 con 0x7fbb7c109d50 2024-09-21T05:48:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.677+0000 7fbb78ff9640 1 -- 172.21.15.28:0/719421776 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74010830 con 0x7fbb7c109d50 2024-09-21T05:48:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.678+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c1a0760 con 0x7fbb7c109d50 2024-09-21T05:48:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.679+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c1a0bd0 con 0x7fbb7c109d50 2024-09-21T05:48:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.683+0000 7fbb8907d740 1 Processor -- start 2024-09-21T05:48:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.684+0000 7fbb78ff9640 1 -- 172.21.15.28:0/719421776 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbb74003ec0 con 0x7fbb7c109d50 2024-09-21T05:48:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.684+0000 7fbb8907d740 1 -- start start 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdd4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdd4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619ccdd5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619ccd496f0 con 0x5619cca062f0 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619ccd38b20 con 0x5619cca0e4d0 2024-09-21T05:48:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.685+0000 7fbb8907d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5619ccd3c930 con 0x5619cca2f1b0 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb78ff9640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 0x7fbb640b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdd4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdd4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44432/0 (socket says 172.21.15.28:44432) 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb7b7fe640 1 -- 172.21.15.28:0/3484825967 learned_addr learned my addr 172.21.15.28:0/3484825967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619ccdd5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdd4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.686+0000 7fbb78ff9640 1 -- 172.21.15.28:0/719421776 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb740e7bb0 con 0x7fbb7c109d50 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 -- 172.21.15.28:0/3484825967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 msgr2=0x5619ccdd4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdd4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 -- 172.21.15.28:0/3484825967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 msgr2=0x5619ccdd4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdd4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 -- 172.21.15.28:0/3484825967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619cccfe530 con 0x5619cca0e4d0 2024-09-21T05:48:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619ccdd5410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c031440 tx=0x7fbb6c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3697dc7afdb967da server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb61ffb640 1 -- 172.21.15.28:0/3484825967 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c035030 con 0x5619cca0e4d0 2024-09-21T05:48:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb61ffb640 1 -- 172.21.15.28:0/3484825967 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6c0314a0 con 0x5619cca0e4d0 2024-09-21T05:48:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb61ffb640 1 -- 172.21.15.28:0/3484825967 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c04e6c0 con 0x5619cca0e4d0 2024-09-21T05:48:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.687+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 0x7fbb640b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 -- 172.21.15.28:0/3484825967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 msgr2=0x5619ccdd5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619ccdd5410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c031440 tx=0x7fbb6c005ef0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb7bfff640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 0x7fbb640b8d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbb740e5920 tx=0x7fbb74009b50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 -- 172.21.15.28:0/3484825967 shutdown_connections 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619ccdd5410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdd4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3484825967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdd4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.689+0000 7fbb8907d740 1 -- 172.21.15.28:0/3484825967 >> 172.21.15.28:0/3484825967 conn(0x5619cca23300 msgr2=0x5619ccdd2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.691+0000 7fbb8907d740 1 -- 172.21.15.28:0/3484825967 shutdown_connections 2024-09-21T05:48:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.691+0000 7fbb8907d740 1 -- 172.21.15.28:0/3484825967 wait complete. 2024-09-21T05:48:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.692+0000 7fbb8907d740 1 -- start start 2024-09-21T05:48:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.692+0000 7fbb8907d740 1 Processor -- start 2024-09-21T05:48:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 0x5619ccde05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdec3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619ccc70ab0 con 0x5619cca0e4d0 2024-09-21T05:48:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619cccde730 con 0x5619cca062f0 2024-09-21T05:48:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5619ccde3100 con 0x5619cca2f1b0 2024-09-21T05:48:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb8907d740 10 client.0 _ll_register_callbacks cb 0x5619ccd326b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.693+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59452/0 (socket says 172.21.15.28:59452) 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 -- 172.21.15.28:0/3267158783 learned_addr learned my addr 172.21.15.28:0/3267158783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdec3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 0x5619ccde05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 -- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 msgr2=0x5619ccdec3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdec3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 -- 172.21.15.28:0/3267158783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 msgr2=0x5619ccde05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 0x5619ccde05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 -- 172.21.15.28:0/3267158783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619cca9a790 con 0x5619cca062f0 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 0x5619ccde05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0096c0 tx=0x7fbb6c031dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=203390c3b1fe52f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.694+0000 7fbb5affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.695+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c035040 con 0x5619cca062f0 2024-09-21T05:48:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.695+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb6c031850 con 0x5619cca062f0 2024-09-21T05:48:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.695+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6c0314a0 con 0x5619cca062f0 2024-09-21T05:48:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.695+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb6c003e90 con 0x5619cca062f0 2024-09-21T05:48:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.695+0000 7fbb8907d740 10 client.25799 fetch_fsmap learned FSMap version 29 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.696+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5619ccb13620 con 0x5619cca062f0 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.696+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6c0314a0 con 0x5619cca062f0 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.696+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb6c0048a0 con 0x5619cca062f0 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 10 client.25799 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 10 client.25799 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 20 client.25799 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cc9edd80 0x5619ccdf1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 4 client.25799 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619ccd4b640 con 0x5619cc9edd80 2024-09-21T05:48:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.697+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cc9edd80 0x5619ccdf1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.699+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cc9edd80 0x5619ccdf1fa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560d0f590ea25997 server_cookie=4bd67059a77b9ac2 in_seq=0 out_seq=0 2024-09-21T05:48:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.699+0000 7fbb5affd640 10 client.25799 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.701+0000 7fbb5affd640 1 -- 172.21.15.28:0/3267158783 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5619ccd4b640 con 0x5619cc9edd80 2024-09-21T05:48:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.701+0000 7fbb5affd640 10 client.25799 handle_command_reply: tid=0 2024-09-21T05:48:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.701+0000 7fbb8907d740 1 client.25799 shutdown 2024-09-21T05:48:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 msgr2=0x5619ccde00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0096c0 tx=0x7fbb6c031dc0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 shutdown_connections 2024-09-21T05:48:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cc9edd80 0x5619ccdf1fa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb5affd640 0 client.25799 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca2f1b0 0x5619ccdec3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca0e4d0 0x5619ccde05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3267158783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca062f0 0x5619ccde00b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 >> 172.21.15.28:0/3267158783 conn(0x5619cca23300 msgr2=0x5619ccdd59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 shutdown_connections 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 1 -- 172.21.15.28:0/3267158783 wait complete. 2024-09-21T05:48:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.702+0000 7fbb8907d740 20 client.25799 trim_cache size 0 max 16384 2024-09-21T05:48:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.732+0000 7fbb8907d740 1 Processor -- start 2024-09-21T05:48:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.732+0000 7fbb8907d740 1 -- start start 2024-09-21T05:48:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 0x5619ccde0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdec360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 0x5619ccde0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 0x5619ccde0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59456/0 (socket says 172.21.15.28:59456) 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb7affd640 1 -- 172.21.15.28:0/2741143048 learned_addr learned my addr 172.21.15.28:0/2741143048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb8907d740 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca0e4d0 0x5619ccde1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.733+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c074bb0 con 0x5619cca062f0 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1b7880 con 0x5619cc9edd80 2024-09-21T05:48:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c075280 con 0x5619cca0e4d0 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb7b7fe640 1 -- 172.21.15.28:0/2741143048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca0e4d0 msgr2=0x5619ccde1ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca0e4d0 0x5619ccde1ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb7b7fe640 1 -- 172.21.15.28:0/2741143048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 msgr2=0x5619ccde0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 0x5619ccde0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.734+0000 7fbb7b7fe640 1 -- 172.21.15.28:0/2741143048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619ccdd7230 con 0x5619cca062f0 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.735+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdec360 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x5619ccd497a0 tx=0x7fbb7000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c40f421a7b5bf87 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.735+0000 7fbb617fa640 1 -- 172.21.15.28:0/2741143048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70005020 con 0x5619cca062f0 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.735+0000 7fbb617fa640 1 -- 172.21.15.28:0/2741143048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb700047a0 con 0x5619cca062f0 2024-09-21T05:48:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.735+0000 7fbb617fa640 1 -- 172.21.15.28:0/2741143048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb700116a0 con 0x5619cca062f0 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 msgr2=0x5619ccdec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdec360 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x5619ccd497a0 tx=0x7fbb7000b390 comp rx=0 tx=0).stop 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 shutdown_connections 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cca0e4d0 0x5619ccde1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdec360 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 --2- 172.21.15.28:0/2741143048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cc9edd80 0x5619ccde0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 >> 172.21.15.28:0/2741143048 conn(0x5619cca23300 msgr2=0x5619ccde15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:25.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 shutdown_connections 2024-09-21T05:48:25.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.736+0000 7fbb8907d740 1 -- 172.21.15.28:0/2741143048 wait complete. 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.737+0000 7fbb8907d740 1 -- start start 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.737+0000 7fbb8907d740 1 Processor -- start 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdf2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:45392/0 (socket says 172.21.15.28:45392) 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619cce546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb7affd640 1 -- 172.21.15.28:0/3077523383 learned_addr learned my addr 172.21.15.28:0/3077523383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1b7180 con 0x5619cca062f0 2024-09-21T05:48:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1b6a80 con 0x5619cca0e4d0 2024-09-21T05:48:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.738+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c072b90 con 0x5619cc9edd80 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619cce546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 -- 172.21.15.28:0/3077523383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 msgr2=0x5619cce546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619cce546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 -- 172.21.15.28:0/3077523383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 msgr2=0x5619ccdf2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdf2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619cce54d50 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb8907d740 10 client.0 _ll_register_callbacks cb 0x7fbb7000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb7affd640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0096c0 tx=0x7fbb6c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d506d3b21381dd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb5affd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb700047a0 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.739+0000 7fbb5affd640 1 -- 172.21.15.28:0/3077523383 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c005df0 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb5affd640 1 -- 172.21.15.28:0/3077523383 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb6c0314a0 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb5affd640 1 -- 172.21.15.28:0/3077523383 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb6c0316a0 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 10 client.25849 fetch_fsmap learned FSMap version 29 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5619cce5bee0 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb5affd640 1 -- 172.21.15.28:0/3077523383 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb6c07b020 con 0x5619cc9edd80 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 10 client.25849 fetch_fsmap finished waiting for FSMap version 29 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 10 client.25849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 20 client.25849 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cca2f1b0 0x5619cce5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 4 client.25849 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:48:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.740+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5619ccd470f0 con 0x5619cca2f1b0 2024-09-21T05:48:25.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.744+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cca2f1b0 0x5619cce5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:25.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.746+0000 7fbb7b7fe640 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cca2f1b0 0x5619cce5e460 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b11fcaee782f1c8 server_cookie=f6f8deca3df5d581 in_seq=0 out_seq=0 2024-09-21T05:48:25.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:25.746+0000 7fbb5affd640 10 client.25849 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:26.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:26 smithi129 ceph-mon[34885]: pgmap v367: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 600 MiB/s wr, 283 op/s 2024-09-21T05:48:26.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:26 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:26.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:26 smithi181 ceph-mon[34819]: pgmap v367: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 600 MiB/s wr, 283 op/s 2024-09-21T05:48:26.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:26 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:26 smithi028 ceph-mon[31227]: pgmap v367: 97 pgs: 97 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 600 MiB/s wr, 283 op/s 2024-09-21T05:48:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:26 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:27.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:27 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:27.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:27 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:27.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:27 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:28.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:28 smithi129 ceph-mon[34885]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 673 MiB/s wr, 1.45k op/s 2024-09-21T05:48:28.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:28 smithi181 ceph-mon[34819]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 673 MiB/s wr, 1.45k op/s 2024-09-21T05:48:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:28 smithi028 ceph-mon[31227]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 673 MiB/s wr, 1.45k op/s 2024-09-21T05:48:30.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:30 smithi129 ceph-mon[34885]: pgmap v369: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 536 MiB/s wr, 2.19k op/s 2024-09-21T05:48:30.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:30 smithi181 ceph-mon[34819]: pgmap v369: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 536 MiB/s wr, 2.19k op/s 2024-09-21T05:48:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:30 smithi028 ceph-mon[31227]: pgmap v369: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 536 MiB/s wr, 2.19k op/s 2024-09-21T05:48:32.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:32 smithi181 ceph-mon[34819]: pgmap v370: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 467 MiB/s wr, 2.40k op/s 2024-09-21T05:48:32.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:32 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:32 smithi028 ceph-mon[31227]: pgmap v370: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 467 MiB/s wr, 2.40k op/s 2024-09-21T05:48:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:32 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:32 smithi129 ceph-mon[34885]: pgmap v370: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 467 MiB/s wr, 2.40k op/s 2024-09-21T05:48:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:32 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:48:34.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:34 smithi181 ceph-mon[34819]: pgmap v371: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 463 MiB/s wr, 4.84k op/s 2024-09-21T05:48:34.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:34 smithi028 ceph-mon[31227]: pgmap v371: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 463 MiB/s wr, 4.84k op/s 2024-09-21T05:48:34.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:34 smithi129 ceph-mon[34885]: pgmap v371: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 463 MiB/s wr, 4.84k op/s 2024-09-21T05:48:36.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:36 smithi181 ceph-mon[34819]: pgmap v372: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 241 MiB/s wr, 5.22k op/s 2024-09-21T05:48:36.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:36 smithi028 ceph-mon[31227]: pgmap v372: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 241 MiB/s wr, 5.22k op/s 2024-09-21T05:48:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:36 smithi129 ceph-mon[34885]: pgmap v372: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 241 MiB/s wr, 5.22k op/s 2024-09-21T05:48:37.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:37 smithi181 ceph-mon[34819]: Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:37.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:37 smithi028 ceph-mon[31227]: Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:37.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:37 smithi129 ceph-mon[34885]: Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:38.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:38 smithi181 ceph-mon[34819]: pgmap v373: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 184 MiB/s wr, 6.80k op/s 2024-09-21T05:48:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:38 smithi028 ceph-mon[31227]: pgmap v373: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 184 MiB/s wr, 6.80k op/s 2024-09-21T05:48:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:38 smithi129 ceph-mon[34885]: pgmap v373: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 184 MiB/s wr, 6.80k op/s 2024-09-21T05:48:40.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:40 smithi181 ceph-mon[34819]: pgmap v374: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 69 MiB/s wr, 6.31k op/s 2024-09-21T05:48:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:40 smithi028 ceph-mon[31227]: pgmap v374: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 69 MiB/s wr, 6.31k op/s 2024-09-21T05:48:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:40 smithi129 ceph-mon[34885]: pgmap v374: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 69 MiB/s wr, 6.31k op/s 2024-09-21T05:48:42.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:42 smithi181 ceph-mon[34819]: pgmap v375: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 39 MiB/s wr, 5.61k op/s 2024-09-21T05:48:42.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:42 smithi028 ceph-mon[31227]: pgmap v375: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 39 MiB/s wr, 5.61k op/s 2024-09-21T05:48:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:42 smithi129 ceph-mon[34885]: pgmap v375: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 39 MiB/s wr, 5.61k op/s 2024-09-21T05:48:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb5affd640 1 -- 172.21.15.28:0/3077523383 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5619ccd470f0 con 0x5619cca2f1b0 2024-09-21T05:48:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb5affd640 10 client.25849 handle_command_reply: tid=0 2024-09-21T05:48:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 client.25849 shutdown 2024-09-21T05:48:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 msgr2=0x5619ccdf2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0096c0 tx=0x7fbb6c005fd0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 shutdown_connections 2024-09-21T05:48:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5619cca2f1b0 0x5619cce5e460 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619cca0e4d0 0x5619cce546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619cca062f0 0x5619ccdf2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 --2- 172.21.15.28:0/3077523383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619cc9edd80 0x5619ccdf2880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb5affd640 0 client.25849 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 >> 172.21.15.28:0/3077523383 conn(0x5619cca23300 msgr2=0x5619ccdd2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.953+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 shutdown_connections 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb8907d740 1 -- 172.21.15.28:0/3077523383 wait complete. 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb8907d740 20 client.25849 trim_cache size 0 max 16384 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:48:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 msgr2=0x7fbb640b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 0x7fbb640b8d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbb740e5920 tx=0x7fbb74009b50 comp rx=0 tx=0).stop 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 msgr2=0x7fbb7c19ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 0x7fbb7c19ff30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400ec50 tx=0x7fbb7400c6a0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 shutdown_connections 2024-09-21T05:48:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb7c109d50 0x7fbb7c19ff30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7c109400 0x7fbb7c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb640b6800 0x7fbb640b8d10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 --2- 172.21.15.28:0/719421776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb7c103ad0 0x7fbb7c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.954+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 >> 172.21.15.28:0/719421776 conn(0x7fbb7c0ff510 msgr2=0x7fbb7c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.955+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 shutdown_connections 2024-09-21T05:48:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:42.955+0000 7fbb82030640 1 -- 172.21.15.28:0/719421776 wait complete. 2024-09-21T05:48:43.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:48:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:44.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:44 smithi028 ceph-mon[31227]: pgmap v376: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 57 MiB/s wr, 7.81k op/s 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.465+0000 7f00f56aa640 1 -- 172.21.15.28:0/1461061550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f01094b0 msgr2=0x7f00f01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.465+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1461061550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f01094b0 0x7f00f01098c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009860 tx=0x7f00e402a360 comp rx=0 tx=0).stop 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.466+0000 7f00f56aa640 1 -- 172.21.15.28:0/1461061550 shutdown_connections 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.467+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1461061550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 0x7f00f0110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.467+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1461061550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f01094b0 0x7f00f01098c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.467+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1461061550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f0103bc0 0x7f00f0103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.467+0000 7f00f56aa640 1 -- 172.21.15.28:0/1461061550 >> 172.21.15.28:0/1461061550 conn(0x7f00f007b910 msgr2=0x7f00f007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.469+0000 7f00f56aa640 1 -- 172.21.15.28:0/1461061550 shutdown_connections 2024-09-21T05:48:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.469+0000 7f00f56aa640 1 -- 172.21.15.28:0/1461061550 wait complete. 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.470+0000 7f00f56aa640 1 Processor -- start 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.470+0000 7f00f56aa640 1 -- start start 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.471+0000 7f00f56aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.471+0000 7f00f56aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01094b0 0x7f00f019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00f56aa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 0x7f00f019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:58002/0 (socket says 172.21.15.28:58002) 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00eeffd640 1 -- 172.21.15.28:0/1553304999 learned_addr learned my addr 172.21.15.28:0/1553304999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0113b00 con 0x7f00f01094b0 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0113980 con 0x7f00f0103bc0 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 0x7f00f019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.472+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0113c80 con 0x7f00f0109e00 2024-09-21T05:48:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01094b0 0x7f00f019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00eeffd640 1 -- 172.21.15.28:0/1553304999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 msgr2=0x7f00f019fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00eeffd640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 0x7f00f019fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00eeffd640 1 -- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01094b0 msgr2=0x7f00f019aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00eeffd640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01094b0 0x7f00f019aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.475+0000 7f00eeffd640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f01a0480 con 0x7f00f0103bc0 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.479+0000 7f00eeffd640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f00e000d950 tx=0x7f00e000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.479+0000 7f00d7fff640 1 -- 172.21.15.28:0/1553304999 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0017070 con 0x7f00f0103bc0 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.480+0000 7f00d7fff640 1 -- 172.21.15.28:0/1553304999 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e0009c30 con 0x7f00f0103bc0 2024-09-21T05:48:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.480+0000 7f00d7fff640 1 -- 172.21.15.28:0/1553304999 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000f620 con 0x7f00f0103bc0 2024-09-21T05:48:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.480+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f01a0770 con 0x7f00f0103bc0 2024-09-21T05:48:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.480+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f01a6ed0 con 0x7f00f0103bc0 2024-09-21T05:48:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.487+0000 7f00fbc1e740 1 Processor -- start 2024-09-21T05:48:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.489+0000 7f00d7fff640 1 -- 172.21.15.28:0/1553304999 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f00e0010460 con 0x7f00f0103bc0 2024-09-21T05:48:44.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.490+0000 7f00fbc1e740 1 -- start start 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce4b2070 0x5565ce859650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce859b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 0x5565ce85a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565ce7cfe50 con 0x5565ce491390 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5565ce7bf280 con 0x5565ce4b2070 2024-09-21T05:48:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.491+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5565ce7c3090 con 0x5565ce4891b0 2024-09-21T05:48:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.492+0000 7f00d7fff640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 0x7f00d80b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.492+0000 7f00d7fff640 1 -- 172.21.15.28:0/1553304999 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f00e00e7240 con 0x7f00f0103bc0 2024-09-21T05:48:44.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.504+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 0x5565ce85a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.504+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 0x5565ce85a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40870/0 (socket says 172.21.15.28:40870) 2024-09-21T05:48:44.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.504+0000 7f00ee7fc640 1 -- 172.21.15.28:0/966873798 learned_addr learned my addr 172.21.15.28:0/966873798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:44.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.504+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce4b2070 0x5565ce859650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.504+0000 7f00eeffd640 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce859b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.505+0000 7f00eeffd640 1 -- 172.21.15.28:0/966873798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce4b2070 msgr2=0x5565ce859650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.505+0000 7f00eeffd640 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce4b2070 0x5565ce859650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.505+0000 7f00eeffd640 1 -- 172.21.15.28:0/966873798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 msgr2=0x5565ce85a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.505+0000 7f00eeffd640 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 0x5565ce85a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.505+0000 7f00eeffd640 1 -- 172.21.15.28:0/966873798 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ce51d650 con 0x5565ce4891b0 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.512+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 0x7f00d80b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.513+0000 7f00eeffd640 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce859b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00e00e49b0 tx=0x7f00e00ac820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e8c9735a6056d22 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.514+0000 7f00d57fa640 1 -- 172.21.15.28:0/966873798 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0020040 con 0x5565ce4891b0 2024-09-21T05:48:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.514+0000 7f00d57fa640 1 -- 172.21.15.28:0/966873798 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e00ad7b0 con 0x5565ce4891b0 2024-09-21T05:48:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.516+0000 7f00fbc1e740 1 -- 172.21.15.28:0/966873798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 msgr2=0x5565ce859b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.516+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce859b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f00e00e49b0 tx=0x7f00e00ac820 comp rx=0 tx=0).stop 2024-09-21T05:48:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.516+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 0x7f00d80b8ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009fd0 tx=0x7f00e8004a80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 -- 172.21.15.28:0/966873798 shutdown_connections 2024-09-21T05:48:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce491390 0x5565ce85a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce859b90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/966873798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce4b2070 0x5565ce859650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 -- 172.21.15.28:0/966873798 >> 172.21.15.28:0/966873798 conn(0x5565ce4a6460 msgr2=0x5565ce857540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 -- 172.21.15.28:0/966873798 shutdown_connections 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.518+0000 7f00fbc1e740 1 -- 172.21.15.28:0/966873798 wait complete. 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.519+0000 7f00fbc1e740 1 -- start start 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.519+0000 7f00fbc1e740 1 Processor -- start 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce86de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce86e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565ce620700 con 0x5565ce4b2070 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5565ce75a110 con 0x5565ce491390 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5565ce867ed0 con 0x5565ce4891b0 2024-09-21T05:48:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.520+0000 7f00fbc1e740 10 client.0 _ll_register_callbacks cb 0x5565ce773990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.523+0000 7f00ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.523+0000 7f00ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40872/0 (socket says 172.21.15.28:40872) 2024-09-21T05:48:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.523+0000 7f00ef7fe640 1 -- 172.21.15.28:0/4206919126 learned_addr learned my addr 172.21.15.28:0/4206919126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.523+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce86e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.523+0000 7f00eeffd640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce86de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.524+0000 7f00ef7fe640 1 -- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 msgr2=0x5565ce86de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.524+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce86de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.524+0000 7f00ef7fe640 1 -- 172.21.15.28:0/4206919126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 msgr2=0x5565ce86e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.524+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce86e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.524+0000 7f00ef7fe640 1 -- 172.21.15.28:0/4206919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ce787c40 con 0x5565ce4b2070 2024-09-21T05:48:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.525+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce86e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:48:44.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.527+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f00e800b4b0 tx=0x7f00e8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362ed76321c2a67e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.527+0000 7f00d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:44.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.528+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8029060 con 0x5565ce4b2070 2024-09-21T05:48:44.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.528+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00e00ad7b0 con 0x5565ce4b2070 2024-09-21T05:48:44.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.528+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f00e801bab0 con 0x5565ce4b2070 2024-09-21T05:48:44.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.529+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f00e801bcb0 con 0x5565ce4b2070 2024-09-21T05:48:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.530+0000 7f00fbc1e740 10 client.16257 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.530+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5565ce88d410 con 0x5565ce4b2070 2024-09-21T05:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e00ad7b0 con 0x5565ce4b2070 2024-09-21T05:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f00e8019410 con 0x5565ce4b2070 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 10 client.16257 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 20 client.16257 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce4709f0 0x5565ce890db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 4 client.16257 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:44.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.531+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565ce7d1da0 con 0x5565ce4709f0 2024-09-21T05:48:44.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.533+0000 7f00eeffd640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce4709f0 0x5565ce890db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.537+0000 7f00eeffd640 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce4709f0 0x5565ce890db0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fc77fd10613ebc8 server_cookie=ff5aabc53622203 in_seq=0 out_seq=0 2024-09-21T05:48:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.537+0000 7f00d2ff5640 10 client.16257 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.540+0000 7f00d2ff5640 1 -- 172.21.15.28:0/4206919126 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5565ce7d1da0 con 0x5565ce4709f0 2024-09-21T05:48:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.540+0000 7f00d2ff5640 10 client.16257 handle_command_reply: tid=0 2024-09-21T05:48:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.540+0000 7f00fbc1e740 1 client.16257 shutdown 2024-09-21T05:48:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 msgr2=0x5565ce873400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f00e800b4b0 tx=0x7f00e8004af0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:44 smithi181 ceph-mon[34819]: pgmap v376: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 57 MiB/s wr, 7.81k op/s 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 shutdown_connections 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce4709f0 0x5565ce890db0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce873400 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce86e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/4206919126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce86de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 >> 172.21.15.28:0/4206919126 conn(0x5565ce4a6460 msgr2=0x5565ce85c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00d2ff5640 0 client.16257 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.541+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 shutdown_connections 2024-09-21T05:48:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.542+0000 7f00fbc1e740 1 -- 172.21.15.28:0/4206919126 wait complete. 2024-09-21T05:48:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.542+0000 7f00fbc1e740 20 client.16257 trim_cache size 0 max 16384 2024-09-21T05:48:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.579+0000 7f00fbc1e740 1 Processor -- start 2024-09-21T05:48:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.579+0000 7f00fbc1e740 1 -- start start 2024-09-21T05:48:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.579+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce85c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.579+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.579+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce86adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:58032/0 (socket says 172.21.15.28:58032) 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00eeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce85c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00eeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce85c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58566/0 (socket says 172.21.15.28:58566) 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce86adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce86adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40884/0 (socket says 172.21.15.28:40884) 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 -- 172.21.15.28:0/725890373 learned_addr learned my addr 172.21.15.28:0/725890373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00f00754d0 con 0x5565ce4b2070 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0077100 con 0x5565ce491390 2024-09-21T05:48:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0075ba0 con 0x5565ce4891b0 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 -- 172.21.15.28:0/725890373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 msgr2=0x5565ce85c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce85c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 -- 172.21.15.28:0/725890373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 msgr2=0x5565ce86adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce86adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.580+0000 7f00ee7fc640 1 -- 172.21.15.28:0/725890373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ce85b940 con 0x5565ce491390 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.581+0000 7f00ee7fc640 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009860 tx=0x7f00e4030c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35512a2df2497da server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.581+0000 7f00d4ff9640 1 -- 172.21.15.28:0/725890373 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4037030 con 0x5565ce491390 2024-09-21T05:48:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.581+0000 7f00d4ff9640 1 -- 172.21.15.28:0/725890373 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e4038040 con 0x5565ce491390 2024-09-21T05:48:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.581+0000 7f00d4ff9640 1 -- 172.21.15.28:0/725890373 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e40052b0 con 0x5565ce491390 2024-09-21T05:48:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.582+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 msgr2=0x5565ce85cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.582+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009860 tx=0x7f00e4030c50 comp rx=0 tx=0).stop 2024-09-21T05:48:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 shutdown_connections 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4b2070 0x5565ce86adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce85cc90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/725890373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4891b0 0x5565ce85c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 >> 172.21.15.28:0/725890373 conn(0x5565ce4a6460 msgr2=0x5565ce86c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 shutdown_connections 2024-09-21T05:48:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 -- 172.21.15.28:0/725890373 wait complete. 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 -- start start 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.583+0000 7f00fbc1e740 1 Processor -- start 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4891b0 0x5565ce875520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce875a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 0x5565ce875fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce875a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00ee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce875a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:58048/0 (socket says 172.21.15.28:58048) 2024-09-21T05:48:44.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00ee7fc640 1 -- 172.21.15.28:0/3410576928 learned_addr learned my addr 172.21.15.28:0/3410576928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:44.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0076970 con 0x5565ce4891b0 2024-09-21T05:48:44.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00f0076270 con 0x5565ce491390 2024-09-21T05:48:44.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00f00734b0 con 0x5565ce4b2070 2024-09-21T05:48:44.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.584+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 0x5565ce875fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.585+0000 7f00fbc1e740 10 client.0 _ll_register_callbacks cb 0x7f00e800bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:44.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 -- 172.21.15.28:0/3410576928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 msgr2=0x5565ce875a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce875a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 -- 172.21.15.28:0/3410576928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4891b0 msgr2=0x5565ce875520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4891b0 0x5565ce875520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ce864900 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00ef7fe640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 0x5565ce875fa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f00e801b8c0 tx=0x7f00e8019720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f712c27204f05a86 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e800ac10 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00e4038040 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.586+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f00e801bab0 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f00e8003a30 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 10 client.25864 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5565ce865b70 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00e4038040 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f00e804d070 con 0x5565ce4b2070 2024-09-21T05:48:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 10 client.25864 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 10 client.25864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 20 client.25864 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce445da0 0x5565ce87c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 4 client.25864 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.587+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5565ce7cd850 con 0x5565ce445da0 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.589+0000 7f00eeffd640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce445da0 0x5565ce87c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.590+0000 7f00d2ff5640 10 client.25864 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00eeffd640 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce445da0 0x5565ce87c7b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99b2bc72e41c807 server_cookie=806e1adcf794e884 in_seq=0 out_seq=0 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:44.597 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:44.598 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:48:44.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00d2ff5640 1 -- 172.21.15.28:0/3410576928 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1510 (crc 0 0 0) 0x5565ce7cd850 con 0x5565ce445da0 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00d2ff5640 10 client.25864 handle_command_reply: tid=0 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00fbc1e740 1 client.25864 shutdown 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 msgr2=0x5565ce875fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.593+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 0x5565ce875fa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f00e801b8c0 tx=0x7f00e8019720 comp rx=0 tx=0).stop 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 shutdown_connections 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5565ce445da0 0x5565ce87c7b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5565ce4b2070 0x5565ce875fa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5565ce491390 0x5565ce875a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 --2- 172.21.15.28:0/3410576928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ce4891b0 0x5565ce875520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 >> 172.21.15.28:0/3410576928 conn(0x5565ce4a6460 msgr2=0x5565ce85c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00d2ff5640 0 client.25864 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 shutdown_connections 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 1 -- 172.21.15.28:0/3410576928 wait complete. 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.594+0000 7f00fbc1e740 20 client.25864 trim_cache size 0 max 16384 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 msgr2=0x7f00d80b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 0x7f00d80b8ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009fd0 tx=0x7f00e8004a80 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 msgr2=0x7f00f019a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f00e000d950 tx=0x7f00e000de10 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 shutdown_connections 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0109e00 0x7f00f019fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f00d80b67d0 0x7f00d80b8ce0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01094b0 0x7f00f019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 --2- 172.21.15.28:0/1553304999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0103bc0 0x7f00f019a510 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:44.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.595+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 >> 172.21.15.28:0/1553304999 conn(0x7f00f007b910 msgr2=0x7f00f01075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:44.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.596+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 shutdown_connections 2024-09-21T05:48:44.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:44.596+0000 7f00f56aa640 1 -- 172.21.15.28:0/1553304999 wait complete. 2024-09-21T05:48:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:44 smithi129 ceph-mon[34885]: pgmap v376: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 57 MiB/s wr, 7.81k op/s 2024-09-21T05:48:45.590 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:48:45.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:48:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:46.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:46 smithi028 ceph-mon[31227]: pgmap v377: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 45 MiB/s wr, 5.75k op/s 2024-09-21T05:48:46.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:46 smithi181 ceph-mon[34819]: pgmap v377: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 45 MiB/s wr, 5.75k op/s 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 -- 172.21.15.28:0/1797985314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 msgr2=0x7f89201087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 --2- 172.21.15.28:0/1797985314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f89201087b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8910009900 tx=0x7f891002a410 comp rx=0 tx=0).stop 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 -- 172.21.15.28:0/1797985314 shutdown_connections 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 --2- 172.21.15.28:0/1797985314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 --2- 172.21.15.28:0/1797985314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f89201087b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 --2- 172.21.15.28:0/1797985314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f8920102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 -- 172.21.15.28:0/1797985314 >> 172.21.15.28:0/1797985314 conn(0x7f8920079730 msgr2=0x7f8920079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 -- 172.21.15.28:0/1797985314 shutdown_connections 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.566+0000 7f8926ba7640 1 -- 172.21.15.28:0/1797985314 wait complete. 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 Processor -- start 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 -- start start 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8920112960 con 0x7f89201083a0 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89201127e0 con 0x7f8920108cf0 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8926ba7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8920112ae0 con 0x7f8920102a60 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f891ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.567+0000 7f8924943640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44042/0 (socket says 172.21.15.28:44042) 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8925144640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:46.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f891ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42488/0 (socket says 172.21.15.28:42488) 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 -- 172.21.15.28:0/4242777736 learned_addr learned my addr 172.21.15.28:0/4242777736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 -- 172.21.15.28:0/4242777736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 msgr2=0x7f892019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 -- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 msgr2=0x7f892019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 -- 172.21.15.28:0/4242777736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89201a03f0 con 0x7f8920102a60 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8925144640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f891ffff640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8924943640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f890800e9f0 tx=0x7f890800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890800cd80 con 0x7f8920102a60 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89201a06e0 con 0x7f8920102a60 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89080054d0 con 0x7f8920102a60 2024-09-21T05:48:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.568+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908010690 con 0x7f8920102a60 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.569+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89201a6e40 con 0x7f8920102a60 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.569+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8920102e70 con 0x7f8920102a60 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.571+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8908004e30 con 0x7f8920102a60 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.572+0000 7f891dffb640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 0x7f890c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.572+0000 7f891ffff640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 0x7f890c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.572+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f89080e7cb0 con 0x7f8920102a60 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.572+0000 7f891ffff640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 0x7f890c0b8b90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f8910031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.575+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f89080ae220 con 0x7f8920102a60 2024-09-21T05:48:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:46 smithi129 ceph-mon[34885]: pgmap v377: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 45 MiB/s wr, 5.75k op/s 2024-09-21T05:48:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.824+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f892019b5b0 con 0x7f8920102a60 2024-09-21T05:48:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.827+0000 7f891dffb640 1 -- 172.21.15.28:0/4242777736 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11994 (secure 0 0 0) 0x7f892019b5b0 con 0x7f8920102a60 2024-09-21T05:48:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:48:46.829 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":31,"btime":"2024-09-21T05:48:31:109481+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:48:31.050757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:48:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 31 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.831+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 msgr2=0x7f890c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.831+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 0x7f890c0b8b90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f8910031400 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.831+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 msgr2=0x7f892019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.831+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f890800e9f0 tx=0x7f890800edf0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 shutdown_connections 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8920108cf0 0x7f892019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f890c0b6680 0x7f890c0b8b90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89201083a0 0x7f892019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 --2- 172.21.15.28:0/4242777736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8920102a60 0x7f892019a480 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 >> 172.21.15.28:0/4242777736 conn(0x7f8920079730 msgr2=0x7f892006b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 shutdown_connections 2024-09-21T05:48:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:46.832+0000 7f8926ba7640 1 -- 172.21.15.28:0/4242777736 wait complete. 2024-09-21T05:48:47.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:47 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4242777736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:48:47.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:47 smithi181 ceph-mon[34819]: Health check update: 2 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:47 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4242777736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:48:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:47 smithi028 ceph-mon[31227]: Health check update: 2 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:47 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4242777736' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:48:47.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:47 smithi129 ceph-mon[34885]: Health check update: 2 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:48:47.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:48:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:48.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:48 smithi028 ceph-mon[31227]: pgmap v378: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 55 MiB/s wr, 6.82k op/s 2024-09-21T05:48:48.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:48 smithi181 ceph-mon[34819]: pgmap v378: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 55 MiB/s wr, 6.82k op/s 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 -- 172.21.15.28:0/3356081128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280109e00 msgr2=0x7fe280110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 --2- 172.21.15.28:0/3356081128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280109e00 0x7fe280110630 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe278009ca0 tx=0x7fe27802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 -- 172.21.15.28:0/3356081128 shutdown_connections 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 --2- 172.21.15.28:0/3356081128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280109e00 0x7fe280110630 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 --2- 172.21.15.28:0/3356081128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe2801098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 --2- 172.21.15.28:0/3356081128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280103bc0 0x7fe280103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.575+0000 7fe288438640 1 -- 172.21.15.28:0/3356081128 >> 172.21.15.28:0/3356081128 conn(0x7fe28007b910 msgr2=0x7fe28007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.576+0000 7fe288438640 1 -- 172.21.15.28:0/3356081128 shutdown_connections 2024-09-21T05:48:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.576+0000 7fe288438640 1 -- 172.21.15.28:0/3356081128 wait complete. 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 Processor -- start 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 -- start start 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe28019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 0x7fe28019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe280113ac0 con 0x7fe280103bc0 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe280113940 con 0x7fe2801094b0 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe288438640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe280113c40 con 0x7fe280109e00 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe2861d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe2861d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42510/0 (socket says 172.21.15.28:42510) 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 0x7fe28019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe2861d4640 1 -- 172.21.15.28:0/2371182986 learned_addr learned my addr 172.21.15.28:0/2371182986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.577+0000 7fe2859d3640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe28019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 -- 172.21.15.28:0/2371182986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 msgr2=0x7fe28019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 0x7fe28019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 -- 172.21.15.28:0/2371182986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 msgr2=0x7fe28019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe28019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 -- 172.21.15.28:0/2371182986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2801a03a0 con 0x7fe280103bc0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2859d3640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe28019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2869d5640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 0x7fe28019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2861d4640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00d950 tx=0x7fe27c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2777fe640 1 -- 172.21.15.28:0/2371182986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c017030 con 0x7fe280103bc0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2801a0690 con 0x7fe280103bc0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2801a6df0 con 0x7fe280103bc0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2777fe640 1 -- 172.21.15.28:0/2371182986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe27c004020 con 0x7fe280103bc0 2024-09-21T05:48:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.578+0000 7fe2777fe640 1 -- 172.21.15.28:0/2371182986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c0047a0 con 0x7fe280103bc0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.581+0000 7fe28e9ac740 1 Processor -- start 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.581+0000 7fe28e9ac740 1 -- start start 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2777fe640 1 -- 172.21.15.28:0/2371182986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe27c004940 con 0x7fe280103bc0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbc2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7fa390 0x56366dbc2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56366db38ea0 con 0x56366d7fa390 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56366db282d0 con 0x56366d7f21b0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56366db2c0e0 con 0x56366d81b070 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2861d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbc2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2859d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7fa390 0x56366dbc2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44054/0 (socket says 172.21.15.28:44054) 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 -- 172.21.15.28:0/3412583870 learned_addr learned my addr 172.21.15.28:0/3412583870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 -- 172.21.15.28:0/3412583870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 msgr2=0x56366dbc2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbc2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 -- 172.21.15.28:0/3412583870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7fa390 msgr2=0x56366dbc2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7fa390 0x56366dbc2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.582+0000 7fe2869d5640 1 -- 172.21.15.28:0/3412583870 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56366d886650 con 0x56366d81b070 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe2869d5640 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe278033040 tx=0x7fe27802f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae722c29d99d0245 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe274d9a640 1 -- 172.21.15.28:0/3412583870 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278031650 con 0x56366d81b070 2024-09-21T05:48:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe274d9a640 1 -- 172.21.15.28:0/3412583870 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278036040 con 0x56366d81b070 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe274d9a640 1 -- 172.21.15.28:0/3412583870 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278040ce0 con 0x56366d81b070 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe2777fe640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 0x7fe26c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe2861d4640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 0x7fe26c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.583+0000 7fe2777fe640 1 -- 172.21.15.28:0/2371182986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe27c0e70c0 con 0x7fe280103bc0 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 -- 172.21.15.28:0/3412583870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 msgr2=0x56366dbc22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe278033040 tx=0x7fe27802f3c0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe2861d4640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 0x7fe26c0b8b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0e4980 tx=0x7fe27c0d0040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 -- 172.21.15.28:0/3412583870 shutdown_connections 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7fa390 0x56366dbc2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbc2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/3412583870 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbc22c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 -- 172.21.15.28:0/3412583870 >> 172.21.15.28:0/3412583870 conn(0x56366d80f460 msgr2=0x56366dbc01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 -- 172.21.15.28:0/3412583870 shutdown_connections 2024-09-21T05:48:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.584+0000 7fe28e9ac740 1 -- 172.21.15.28:0/3412583870 wait complete. 2024-09-21T05:48:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 -- start start 2024-09-21T05:48:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 Processor -- start 2024-09-21T05:48:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 0x56366dbcd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56366d989700 con 0x56366d81b070 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56366dac3160 con 0x56366d7fa390 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56366dbd08c0 con 0x56366d7f21b0 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe28e9ac740 10 client.0 _ll_register_callbacks cb 0x56366dadc9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.585+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 0x56366dbcd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 0x56366dbcd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44068/0 (socket says 172.21.15.28:44068) 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2869d5640 1 -- 172.21.15.28:0/855746252 learned_addr learned my addr 172.21.15.28:0/855746252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2869d5640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 -- 172.21.15.28:0/855746252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 msgr2=0x56366dbcd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 0x56366dbcd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 -- 172.21.15.28:0/855746252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 msgr2=0x56366dbce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 -- 172.21.15.28:0/855746252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56366daf0c90 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe2859d3640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcde00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe28006a790 tx=0x7fe27000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94d458cdaf0b125 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270012110 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe270004050 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.586+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe278036040 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2700122b0 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 10 client.25838 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56366dbdc550 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278036040 con 0x56366d7fa390 2024-09-21T05:48:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fe270023020 con 0x56366d7fa390 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 10 client.25838 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 10 client.25838 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 20 client.25838 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7d99f0 0x56366dbdfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 4 client.25838 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.587+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56366db3adf0 con 0x56366d7d99f0 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.588+0000 7fe2869d5640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7d99f0 0x56366dbdfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.588+0000 7fe2869d5640 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7d99f0 0x56366dbdfa70 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dcdc8430af66bc34 server_cookie=e51803a882d40556 in_seq=0 out_seq=0 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.588+0000 7fe269ffb640 10 client.25838 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe269ffb640 1 -- 172.21.15.28:0/855746252 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56366db3adf0 con 0x56366d7d99f0 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe269ffb640 10 client.25838 handle_command_reply: tid=0 2024-09-21T05:48:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 client.25838 shutdown 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 msgr2=0x56366dbcde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcde00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe28006a790 tx=0x7fe27000f790 comp rx=0 tx=0).stop 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 shutdown_connections 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7d99f0 0x56366dbdfa70 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcde00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe269ffb640 0 client.25838 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/855746252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7f21b0 0x56366dbcd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 >> 172.21.15.28:0/855746252 conn(0x56366d80f460 msgr2=0x56366dbc5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 shutdown_connections 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 1 -- 172.21.15.28:0/855746252 wait complete. 2024-09-21T05:48:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.589+0000 7fe28e9ac740 20 client.25838 trim_cache size 0 max 16384 2024-09-21T05:48:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 Processor -- start 2024-09-21T05:48:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 -- start start 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbcaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ad470 con 0x56366d7f21b0 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801af0a0 con 0x56366d7fa390 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801adb40 con 0x56366d81b070 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42530/0 (socket says 172.21.15.28:42530) 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2859d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 -- 172.21.15.28:0/414112447 learned_addr learned my addr 172.21.15.28:0/414112447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbcaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 -- 172.21.15.28:0/414112447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 msgr2=0x56366dbcaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbcaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.603+0000 7fe2869d5640 1 -- 172.21.15.28:0/414112447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 msgr2=0x56366dbcaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe2869d5640 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe2869d5640 1 -- 172.21.15.28:0/414112447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56366dbc4820 con 0x56366d7f21b0 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe2859d3640 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe2869d5640 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe278040890 tx=0x7fe278009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67d56ec83b981f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe26bfff640 1 -- 172.21.15.28:0/414112447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278031710 con 0x56366d7f21b0 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe26bfff640 1 -- 172.21.15.28:0/414112447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278036070 con 0x56366d7f21b0 2024-09-21T05:48:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.604+0000 7fe26bfff640 1 -- 172.21.15.28:0/414112447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278004040 con 0x56366d7f21b0 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- 172.21.15.28:0/414112447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 msgr2=0x56366dbc5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe278040890 tx=0x7fe278009450 comp rx=0 tx=0).stop 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- 172.21.15.28:0/414112447 shutdown_connections 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d81b070 0x56366dbcaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7fa390 0x56366dbcaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/414112447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d7f21b0 0x56366dbc5ad0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- 172.21.15.28:0/414112447 >> 172.21.15.28:0/414112447 conn(0x56366d80f460 msgr2=0x56366dbc53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- 172.21.15.28:0/414112447 shutdown_connections 2024-09-21T05:48:48.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- 172.21.15.28:0/414112447 wait complete. 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.605+0000 7fe28e9ac740 1 -- start start 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 Processor -- start 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbd1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbd27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ae910 con 0x56366d81b070 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ae210 con 0x56366d7f21b0 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ab450 con 0x56366d7fa390 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe28e9ac740 10 client.0 _ll_register_callbacks cb 0x7fe270009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44086/0 (socket says 172.21.15.28:44086) 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 -- 172.21.15.28:0/82701690 learned_addr learned my addr 172.21.15.28:0/82701690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2869d5640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbd1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2869d5640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbd27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 -- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 msgr2=0x56366dbd1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbd1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 -- 172.21.15.28:0/82701690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 msgr2=0x56366dbd27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbd27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.606+0000 7fe2859d3640 1 -- 172.21.15.28:0/82701690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56366dc49680 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe2859d3640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe28006a790 tx=0x7fe270025cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2711efa6f8d266a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700128a0 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe2700052c0 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe278036070 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2700054e0 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe28e9ac740 10 client.25888 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56366dc4a900 con 0x56366d7fa390 2024-09-21T05:48:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.607+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278036070 con 0x56366d7fa390 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fe270035070 con 0x56366d7fa390 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 10 client.25888 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 10 client.25888 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 20 client.25888 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7aeda0 0x56366dc4d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 4 client.25888 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56366db368a0 con 0x56366d7aeda0 2024-09-21T05:48:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe2869d5640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7aeda0 0x56366dc4d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe269ffb640 10 client.25888 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.608+0000 7fe2869d5640 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7aeda0 0x56366dc4d110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=24d5e95b9c3ff008 server_cookie=e250c6ecc1284024 in_seq=0 out_seq=0 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe269ffb640 1 -- 172.21.15.28:0/82701690 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56366db368a0 con 0x56366d7aeda0 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe269ffb640 10 client.25888 handle_command_reply: tid=0 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 client.25888 shutdown 2024-09-21T05:48:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 msgr2=0x56366dbd2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe28006a790 tx=0x7fe270025cd0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 shutdown_connections 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x56366d7aeda0 0x56366dc4d110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56366d81b070 0x56366dbd27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56366d7fa390 0x56366dbd2260 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe269ffb640 0 client.25888 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 --2- 172.21.15.28:0/82701690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56366d7f21b0 0x56366dbd1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 >> 172.21.15.28:0/82701690 conn(0x56366d80f460 msgr2=0x56366dbc53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 shutdown_connections 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 1 -- 172.21.15.28:0/82701690 wait complete. 2024-09-21T05:48:48.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.609+0000 7fe28e9ac740 20 client.25888 trim_cache size 0 max 16384 2024-09-21T05:48:48.616 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:48.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 msgr2=0x7fe26c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 0x7fe26c0b8b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0e4980 tx=0x7fe27c0d0040 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 msgr2=0x7fe28019a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00d950 tx=0x7fe27c00de10 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 shutdown_connections 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe280109e00 0x7fe28019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2801094b0 0x7fe28019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe26c0b6680 0x7fe26c0b8b90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 --2- 172.21.15.28:0/2371182986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280103bc0 0x7fe28019a430 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 >> 172.21.15.28:0/2371182986 conn(0x7fe28007b910 msgr2=0x7fe2801074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 shutdown_connections 2024-09-21T05:48:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:48.610+0000 7fe288438640 1 -- 172.21.15.28:0/2371182986 wait complete. 2024-09-21T05:48:48.618 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:48 smithi129 ceph-mon[34885]: pgmap v378: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 55 MiB/s wr, 6.82k op/s 2024-09-21T05:48:49.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:48:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:48:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:48:49.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:48:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:48:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:48:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:48:49.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:48:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:48:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:48:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:48:49.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:48:49.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:48:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:50 smithi028 ceph-mon[31227]: pgmap v379: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 51 MiB/s wr, 6.32k op/s 2024-09-21T05:48:50.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:50 smithi181 ceph-mon[34819]: pgmap v379: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 51 MiB/s wr, 6.32k op/s 2024-09-21T05:48:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:50 smithi129 ceph-mon[34885]: pgmap v379: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 51 MiB/s wr, 6.32k op/s 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- 172.21.15.28:0/2911405268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168108d20 msgr2=0x7f516810f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2911405268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168108d20 0x7f516810f550 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5160009ca0 tx=0x7f516002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- 172.21.15.28:0/2911405268 shutdown_connections 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2911405268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168108d20 0x7f516810f550 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2911405268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 0x7f51681087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2911405268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168102a90 0x7f5168102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- 172.21.15.28:0/2911405268 >> 172.21.15.28:0/2911405268 conn(0x7f5168079750 msgr2=0x7f5168079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- 172.21.15.28:0/2911405268 shutdown_connections 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- 172.21.15.28:0/2911405268 wait complete. 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 Processor -- start 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.676+0000 7f516ebbd640 1 -- start start 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 0x7f516819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168108d20 0x7f516819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112990 con 0x7f5168102a90 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112810 con 0x7f5168108d20 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516ebbd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112b10 con 0x7f51681083d0 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f5167fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 0x7f516819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42562/0 (socket says 172.21.15.28:42562) 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 -- 172.21.15.28:0/2193213131 learned_addr learned my addr 172.21.15.28:0/2193213131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516d15a640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168108d20 0x7f516819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 -- 172.21.15.28:0/2193213131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 msgr2=0x7f516819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 0x7f516819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 -- 172.21.15.28:0/2193213131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168108d20 msgr2=0x7f516819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168108d20 0x7f516819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.677+0000 7f516c959640 1 -- 172.21.15.28:0/2193213131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51681a0440 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f516c959640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f515c00d950 tx=0x7f515c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f5165ffb640 1 -- 172.21.15.28:0/2193213131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c017030 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51681a0730 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f5165ffb640 1 -- 172.21.15.28:0/2193213131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f515c004020 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f5165ffb640 1 -- 172.21.15.28:0/2193213131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c0047a0 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.678+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51681a6e90 con 0x7f5168102a90 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 Processor -- start 2024-09-21T05:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5165ffb640 1 -- 172.21.15.28:0/2193213131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f515c009a30 con 0x7f5168102a90 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 -- start start 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe88651b0 0x55dfe8c35800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c35d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfe8babea0 con 0x55dfe88651b0 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dfe8b9b2d0 con 0x55dfe888e070 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5175131740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dfe8b9f0e0 con 0x55dfe886d390 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f516d15a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f516d15a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:49134/0 (socket says 172.21.15.28:49134) 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f516d15a640 1 -- 172.21.15.28:0/466579400 learned_addr learned my addr 172.21.15.28:0/466579400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f516c959640 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe88651b0 0x55dfe8c35800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.681+0000 7f5167fff640 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c35d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 -- 172.21.15.28:0/466579400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 msgr2=0x55dfe8c35d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c35d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 -- 172.21.15.28:0/466579400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe88651b0 msgr2=0x55dfe8c35800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe88651b0 0x55dfe8c35800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 -- 172.21.15.28:0/466579400 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfe88f9650 con 0x55dfe888e070 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f5165ffb640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 0x7f51580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516c959640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 0x7f51580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f516d15a640 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f516002f4c0 tx=0x7f5160009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9d0f34d62ea6cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f5165ffb640 1 -- 172.21.15.28:0/2193213131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f515c0e70b0 con 0x7f5168102a90 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f5152ffd640 1 -- 172.21.15.28:0/466579400 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160031cf0 con 0x55dfe888e070 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f5152ffd640 1 -- 172.21.15.28:0/466579400 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5160036030 con 0x55dfe888e070 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.682+0000 7f5152ffd640 1 -- 172.21.15.28:0/466579400 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600042a0 con 0x55dfe888e070 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 -- 172.21.15.28:0/466579400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 msgr2=0x55dfe8c352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f516c959640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 0x7f51580b8b90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f515c0e48d0 tx=0x7f515c0eac00 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f516002f4c0 tx=0x7f5160009800 comp rx=0 tx=0).stop 2024-09-21T05:48:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 -- 172.21.15.28:0/466579400 shutdown_connections 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c35d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe88651b0 0x55dfe8c35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 --2- 172.21.15.28:0/466579400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe888e070 0x55dfe8c352c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 -- 172.21.15.28:0/466579400 >> 172.21.15.28:0/466579400 conn(0x55dfe8882460 msgr2=0x55dfe8c331b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 -- 172.21.15.28:0/466579400 shutdown_connections 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.683+0000 7f5175131740 1 -- 172.21.15.28:0/466579400 wait complete. 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 -- start start 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 Processor -- start 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe888e070 0x55dfe8c41460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfe89fc700 con 0x55dfe888e070 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dfe8b36160 con 0x55dfe886d390 2024-09-21T05:48:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dfe8c438c0 con 0x55dfe88651b0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.684+0000 7f5175131740 10 client.0 _ll_register_callbacks cb 0x55dfe8b4f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f516d15a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe888e070 0x55dfe8c41460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:49140/0 (socket says 172.21.15.28:49140) 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 -- 172.21.15.28:0/763118291 learned_addr learned my addr 172.21.15.28:0/763118291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f516d15a640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 -- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 msgr2=0x55dfe8c409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 -- 172.21.15.28:0/763118291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe888e070 msgr2=0x55dfe8c41460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe888e070 0x55dfe8c41460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 -- 172.21.15.28:0/763118291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfe8b63c90 con 0x55dfe886d390 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5167fff640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5154012700 tx=0x7f5154012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15469fe5ca3d41f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5147fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515400f8c0 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f515400fa60 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.685+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5160036030 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f515400fc60 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 10 client.25856 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dfe8c4f5d0 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5160036030 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f5154023020 con 0x55dfe886d390 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 10 client.25856 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 10 client.25856 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 20 client.25856 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe884c9f0 0x55dfe8c52a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 4 client.25856 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfe8baddf0 con 0x55dfe884c9f0 2024-09-21T05:48:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.686+0000 7f516d15a640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe884c9f0 0x55dfe8c52a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.687+0000 7f5147fff640 10 client.25856 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.687+0000 7f516d15a640 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe884c9f0 0x55dfe8c52a90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e0a18aa798a98378 server_cookie=6c6f4dda4d04c67c in_seq=0 out_seq=0 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.687+0000 7f5147fff640 1 -- 172.21.15.28:0/763118291 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dfe8baddf0 con 0x55dfe884c9f0 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.687+0000 7f5147fff640 10 client.25856 handle_command_reply: tid=0 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 client.25856 shutdown 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 msgr2=0x55dfe8c40f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5154012700 tx=0x7f5154012bc0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 shutdown_connections 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe884c9f0 0x55dfe8c52a90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe888e070 0x55dfe8c41460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe886d390 0x55dfe8c40f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5147fff640 0 client.25856 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 --2- 172.21.15.28:0/763118291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 >> 172.21.15.28:0/763118291 conn(0x55dfe8882460 msgr2=0x55dfe8c38450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 shutdown_connections 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 1 -- 172.21.15.28:0/763118291 wait complete. 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.688+0000 7f5175131740 20 client.25856 trim_cache size 0 max 16384 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.701+0000 7f5175131740 1 Processor -- start 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 -- start start 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe88651b0 0x55dfe8c425f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c4cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51681af6e0 con 0x55dfe884c9f0 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51681b1310 con 0x55dfe88651b0 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5175131740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51681afdb0 con 0x55dfe886d390 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42596/0 (socket says 172.21.15.28:42596) 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 -- 172.21.15.28:0/2885209195 learned_addr learned my addr 172.21.15.28:0/2885209195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f5167fff640 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe88651b0 0x55dfe8c425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c4cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 -- 172.21.15.28:0/2885209195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 msgr2=0x55dfe8c4cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c4cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 -- 172.21.15.28:0/2885209195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe88651b0 msgr2=0x55dfe8c425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe88651b0 0x55dfe8c425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 -- 172.21.15.28:0/2885209195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfe8c3eec0 con 0x55dfe884c9f0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.702+0000 7f516d15a640 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5160009870 tx=0x7f516002f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b890701b4327ae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.703+0000 7f51527fc640 1 -- 172.21.15.28:0/2885209195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160004b40 con 0x55dfe884c9f0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.703+0000 7f51527fc640 1 -- 172.21.15.28:0/2885209195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5160036070 con 0x55dfe884c9f0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.703+0000 7f51527fc640 1 -- 172.21.15.28:0/2885209195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160031750 con 0x55dfe884c9f0 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.703+0000 7f5175131740 1 -- 172.21.15.28:0/2885209195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 msgr2=0x55dfe8c36cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.703+0000 7f5175131740 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5160009870 tx=0x7f516002f3a0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 -- 172.21.15.28:0/2885209195 shutdown_connections 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe886d390 0x55dfe8c4cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe88651b0 0x55dfe8c425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 --2- 172.21.15.28:0/2885209195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe884c9f0 0x55dfe8c36cb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 -- 172.21.15.28:0/2885209195 >> 172.21.15.28:0/2885209195 conn(0x55dfe8882460 msgr2=0x55dfe8c41ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 -- 172.21.15.28:0/2885209195 shutdown_connections 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 -- 172.21.15.28:0/2885209195 wait complete. 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 -- start start 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.704+0000 7f5175131740 1 Processor -- start 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c45830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe886d390 0x55dfe8cbc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51681b0b80 con 0x55dfe886d390 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51681b0480 con 0x55dfe884c9f0 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51681ad6c0 con 0x55dfe88651b0 2024-09-21T05:48:50.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5175131740 10 client.0 _ll_register_callbacks cb 0x7f5154009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe886d390 0x55dfe8cbc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5167fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c45830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:49158/0 (socket says 172.21.15.28:49158) 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5167fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c45830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44140/0 (socket says 172.21.15.28:44140) 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 -- 172.21.15.28:0/2359676936 learned_addr learned my addr 172.21.15.28:0/2359676936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 -- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 msgr2=0x55dfe8c45830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c45830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 -- 172.21.15.28:0/2359676936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe886d390 msgr2=0x55dfe8cbc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe886d390 0x55dfe8cbc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 -- 172.21.15.28:0/2359676936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfe8cbca40 con 0x55dfe884c9f0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f516d15a640 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5160009ca0 tx=0x7f516002a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aeec99dfa327ef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5147fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:50.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160004b40 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.705+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5160004ce0 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5160036070 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5160031af0 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 10 client.25862 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dfe8cbdca0 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5160036070 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f5160043070 con 0x55dfe884c9f0 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 10 client.25862 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 10 client.25862 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 20 client.25862 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe888e070 0x55dfe8cc0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 4 client.25862 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfe8ba98a0 con 0x55dfe888e070 2024-09-21T05:48:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.706+0000 7f5167fff640 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe888e070 0x55dfe8cc0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5167fff640 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe888e070 0x55dfe8cc0010 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e168a2c9b3b13a7 server_cookie=30826e0c4404ea2c in_seq=0 out_seq=0 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5147fff640 10 client.25862 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5147fff640 1 -- 172.21.15.28:0/2359676936 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dfe8ba98a0 con 0x55dfe888e070 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5147fff640 10 client.25862 handle_command_reply: tid=0 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 client.25862 shutdown 2024-09-21T05:48:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 msgr2=0x55dfe8c452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5160009ca0 tx=0x7f516002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 shutdown_connections 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55dfe888e070 0x55dfe8cc0010 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfe886d390 0x55dfe8cbc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfe88651b0 0x55dfe8c45830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 --2- 172.21.15.28:0/2359676936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfe884c9f0 0x55dfe8c452f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5147fff640 0 client.25862 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 >> 172.21.15.28:0/2359676936 conn(0x55dfe8882460 msgr2=0x55dfe8c36a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 shutdown_connections 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 1 -- 172.21.15.28:0/2359676936 wait complete. 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.707+0000 7f5175131740 20 client.25862 trim_cache size 0 max 16384 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 msgr2=0x7f51580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 0x7f51580b8b90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f515c0e48d0 tx=0x7f515c0eac00 comp rx=0 tx=0).stop 2024-09-21T05:48:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 msgr2=0x7f516819a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f515c00d950 tx=0x7f515c00de10 comp rx=0 tx=0).stop 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 shutdown_connections 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5168108d20 0x7f516819fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51681083d0 0x7f516819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f51580b6680 0x7f51580b8b90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 --2- 172.21.15.28:0/2193213131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5168102a90 0x7f516819a4d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 >> 172.21.15.28:0/2193213131 conn(0x7f5168079750 msgr2=0x7f516806bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 shutdown_connections 2024-09-21T05:48:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:50.708+0000 7f516ebbd640 1 -- 172.21.15.28:0/2193213131 wait complete. 2024-09-21T05:48:51.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:48:52.230 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:52.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:52 smithi028 ceph-mon[31227]: pgmap v380: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:48:52.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:52 smithi181 ceph-mon[34819]: pgmap v380: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:48:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:52 smithi129 ceph-mon[34885]: pgmap v380: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:48:52.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- 172.21.15.28:0/905709755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 msgr2=0x7fdb6c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 --2- 172.21.15.28:0/905709755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c102ef0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009900 tx=0x7fdb5c02a410 comp rx=0 tx=0).stop 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- 172.21.15.28:0/905709755 shutdown_connections 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 --2- 172.21.15.28:0/905709755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 0x7fdb6c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 --2- 172.21.15.28:0/905709755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 --2- 172.21.15.28:0/905709755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c102ef0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- 172.21.15.28:0/905709755 >> 172.21.15.28:0/905709755 conn(0x7fdb6c079760 msgr2=0x7fdb6c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- 172.21.15.28:0/905709755 shutdown_connections 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- 172.21.15.28:0/905709755 wait complete. 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 Processor -- start 2024-09-21T05:48:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.728+0000 7fdb71f17640 1 -- start start 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c19b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 0x7fdb6c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c112990 con 0x7fdb6c102ae0 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c112810 con 0x7fdb6c1083d0 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb71f17640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c112b10 con 0x7fdb6c108d20 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42628/0 (socket says 172.21.15.28:42628) 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 -- 172.21.15.28:0/4193439323 learned_addr learned my addr 172.21.15.28:0/4193439323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 0x7fdb6c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c19b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 -- 172.21.15.28:0/4193439323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 msgr2=0x7fdb6c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 0x7fdb6c19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 -- 172.21.15.28:0/4193439323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 msgr2=0x7fdb6c19b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c19b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb6b7fe640 1 -- 172.21.15.28:0/4193439323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c1a6590 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.729+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c19b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005940 tx=0x7fdb5c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb697fa640 1 -- 172.21.15.28:0/4193439323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c03f030 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb697fa640 1 -- 172.21.15.28:0/4193439323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5c003aa0 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb697fa640 1 -- 172.21.15.28:0/4193439323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c035070 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6c1a6820 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.730+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6c1a6d00 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.732+0000 7fdb7848b740 1 Processor -- start 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 -- start start 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb697fa640 1 -- 172.21.15.28:0/4193439323 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fdb5c042030 con 0x7fdb6c102ae0 2024-09-21T05:48:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x5622864606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286461120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622863d6ea0 con 0x5622860901b0 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5622863c62d0 con 0x562286098390 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb7848b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5622863ca0e0 con 0x5622860b9070 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x5622864606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42630/0 (socket says 172.21.15.28:42630) 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6b7fe640 1 -- 172.21.15.28:0/1430938641 learned_addr learned my addr 172.21.15.28:0/1430938641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286461120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 -- 172.21.15.28:0/1430938641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 msgr2=0x562286461120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286461120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 -- 172.21.15.28:0/1430938641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 msgr2=0x562286460be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6bfff640 1 -- 172.21.15.28:0/1430938641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562286124650 con 0x5622860b9070 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.733+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x5622864606a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009ee0 tx=0x7fdb64010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89eba1abf1da18bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb697fa640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 0x7fdb580b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 0x7fdb580b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb61ffb640 1 -- 172.21.15.28:0/1430938641 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64016070 con 0x5622860b9070 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb61ffb640 1 -- 172.21.15.28:0/1430938641 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb64004d20 con 0x5622860b9070 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb61ffb640 1 -- 172.21.15.28:0/1430938641 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64012720 con 0x5622860b9070 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.734+0000 7fdb697fa640 1 -- 172.21.15.28:0/4193439323 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdb5c107df0 con 0x7fdb6c102ae0 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- 172.21.15.28:0/1430938641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 msgr2=0x5622864606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb6b7fe640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 0x7fdb580b8bc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0ca230 tx=0x7fdb5c10b940 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x5622864606a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009ee0 tx=0x7fdb64010740 comp rx=0 tx=0).stop 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- 172.21.15.28:0/1430938641 shutdown_connections 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286461120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286460be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1430938641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x5622864606a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- 172.21.15.28:0/1430938641 >> 172.21.15.28:0/1430938641 conn(0x5622860ad460 msgr2=0x56228645e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- 172.21.15.28:0/1430938641 shutdown_connections 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- 172.21.15.28:0/1430938641 wait complete. 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 -- start start 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.735+0000 7fdb7848b740 1 Processor -- start 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286470690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x56228647c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562286227700 con 0x5622860901b0 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562286361160 con 0x562286098390 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5622864731b0 con 0x5622860b9070 2024-09-21T05:48:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb7848b740 10 client.0 _ll_register_callbacks cb 0x56228637a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286470690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42638/0 (socket says 172.21.15.28:42638) 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb6bfff640 1 -- 172.21.15.28:0/1207893924 learned_addr learned my addr 172.21.15.28:0/1207893924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x56228647c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 msgr2=0x56228647c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x56228647c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1207893924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 msgr2=0x562286470150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1207893924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56228638ec90 con 0x562286098390 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.736+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286470690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c06b680 tx=0x7fdb54012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a229da8d4af20749 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54004320 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdb540119a0 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb64004d20 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdb540044c0 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb7848b740 10 client.25874 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56228647ef40 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb64004d20 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.737+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fdb54023020 con 0x562286098390 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 10 client.25874 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 10 client.25874 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 20 client.25874 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860779f0 0x5622864823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 4 client.25874 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622863d8df0 con 0x5622860779f0 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860779f0 0x5622864823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb4affd640 10 client.25874 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.738+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860779f0 0x5622864823f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67a1624f89f6cc06 server_cookie=96317f373e6a9226 in_seq=0 out_seq=0 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb4affd640 1 -- 172.21.15.28:0/1207893924 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5622863d8df0 con 0x5622860779f0 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb4affd640 10 client.25874 handle_command_reply: tid=0 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 client.25874 shutdown 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 msgr2=0x562286470690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286470690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c06b680 tx=0x7fdb54012b60 comp rx=0 tx=0).stop 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 shutdown_connections 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860779f0 0x5622864823f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860b9070 0x56228647c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562286098390 0x562286470690 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 --2- 172.21.15.28:0/1207893924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860901b0 0x562286470150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb4affd640 0 client.25874 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 >> 172.21.15.28:0/1207893924 conn(0x5622860ad460 msgr2=0x562286463830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 shutdown_connections 2024-09-21T05:48:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 1 -- 172.21.15.28:0/1207893924 wait complete. 2024-09-21T05:48:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.739+0000 7fdb7848b740 20 client.25874 trim_cache size 0 max 16384 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 Processor -- start 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 -- start start 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860ad460 0x56228646f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 0x56228645dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1ad540 con 0x562286098390 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1af170 con 0x5622860901b0 2024-09-21T05:48:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb7848b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1adc10 con 0x5622860ad460 2024-09-21T05:48:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860ad460 0x56228646f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 0x56228645dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.753+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:49198/0 (socket says 172.21.15.28:49198) 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 -- 172.21.15.28:0/236722136 learned_addr learned my addr 172.21.15.28:0/236722136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 -- 172.21.15.28:0/236722136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860ad460 msgr2=0x56228646f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860ad460 0x56228646f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 -- 172.21.15.28:0/236722136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 msgr2=0x56228645dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 0x56228645dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 -- 172.21.15.28:0/236722136 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622864708c0 con 0x5622860901b0 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 0x56228645dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009d30 tx=0x7fdb54011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb9687cbc104792c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb617fa640 1 -- 172.21.15.28:0/236722136 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54025690 con 0x5622860901b0 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb617fa640 1 -- 172.21.15.28:0/236722136 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54025e30 con 0x5622860901b0 2024-09-21T05:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.754+0000 7fdb617fa640 1 -- 172.21.15.28:0/236722136 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb540052a0 con 0x5622860901b0 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 -- 172.21.15.28:0/236722136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 msgr2=0x562286461de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009d30 tx=0x7fdb54011810 comp rx=0 tx=0).stop 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 -- 172.21.15.28:0/236722136 shutdown_connections 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562286098390 0x56228645dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x562286461de0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 --2- 172.21.15.28:0/236722136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5622860ad460 0x56228646f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 -- 172.21.15.28:0/236722136 >> 172.21.15.28:0/236722136 conn(0x56228647c960 msgr2=0x5622864616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 -- 172.21.15.28:0/236722136 shutdown_connections 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.755+0000 7fdb7848b740 1 -- 172.21.15.28:0/236722136 wait complete. 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 -- start start 2024-09-21T05:48:52.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 Processor -- start 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x5622864745e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562286098390 0x562286474b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1ae9e0 con 0x5622860ad460 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1ae2e0 con 0x5622860901b0 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1ab520 con 0x562286098390 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb7848b740 10 client.0 _ll_register_callbacks cb 0x7fdb54009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42654/0 (socket says 172.21.15.28:42654) 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.756+0000 7fdb6bfff640 1 -- 172.21.15.28:0/2859945353 learned_addr learned my addr 172.21.15.28:0/2859945353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562286098390 0x562286474b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x5622864745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 -- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562286098390 msgr2=0x562286474b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562286098390 0x562286474b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 -- 172.21.15.28:0/2859945353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 msgr2=0x5622864745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x5622864745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 -- 172.21.15.28:0/2859945353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622864ebfb0 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb6bfff640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009e00 tx=0x7fdb64010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58414560e960122a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb4affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64016040 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdb640054e0 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb54025e30 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdb640056e0 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.757+0000 7fdb7848b740 10 client.16320 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5622864ed1f0 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54025e30 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fdb64028040 con 0x5622860ad460 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 10 client.16320 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 10 client.16320 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:48:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 20 client.16320 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860b9070 0x5622864ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 4 client.16320 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622863d48a0 con 0x5622860b9070 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.758+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860b9070 0x5622864ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860b9070 0x5622864ef7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b453f18c94e17e46 server_cookie=7d656ad024fb5aa4 in_seq=0 out_seq=0 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb4affd640 10 client.16320 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb4affd640 1 -- 172.21.15.28:0/2859945353 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622863d48a0 con 0x5622860b9070 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb4affd640 10 client.16320 handle_command_reply: tid=0 2024-09-21T05:48:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 client.16320 shutdown 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 msgr2=0x562286475060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009e00 tx=0x7fdb64010740 comp rx=0 tx=0).stop 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 shutdown_connections 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5622860b9070 0x5622864ef7a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622860ad460 0x562286475060 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562286098390 0x562286474b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 --2- 172.21.15.28:0/2859945353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5622860901b0 0x5622864745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 >> 172.21.15.28:0/2859945353 conn(0x56228647c960 msgr2=0x562286462090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb4affd640 0 client.16320 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 shutdown_connections 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.759+0000 7fdb7848b740 1 -- 172.21.15.28:0/2859945353 wait complete. 2024-09-21T05:48:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb7848b740 20 client.16320 trim_cache size 0 max 16384 2024-09-21T05:48:52.766 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:52.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 msgr2=0x7fdb580b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 0x7fdb580b8bc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0ca230 tx=0x7fdb5c10b940 comp rx=0 tx=0).stop 2024-09-21T05:48:52.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 msgr2=0x7fdb6c19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:52.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005940 tx=0x7fdb5c004270 comp rx=0 tx=0).stop 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 shutdown_connections 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb6c108d20 0x7fdb6c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb6c1083d0 0x7fdb6c19b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdb580b66b0 0x7fdb580b8bc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 --2- 172.21.15.28:0/4193439323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb6c102ae0 0x7fdb6c19b300 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 >> 172.21.15.28:0/4193439323 conn(0x7fdb6c079760 msgr2=0x7fdb6c101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 shutdown_connections 2024-09-21T05:48:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:52.760+0000 7fdb71f17640 1 -- 172.21.15.28:0/4193439323 wait complete. 2024-09-21T05:48:53.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:48:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 -- 172.21.15.28:0/1217520196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 msgr2=0x7f94201087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 --2- 172.21.15.28:0/1217520196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f94201087f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9414009900 tx=0x7f941402a410 comp rx=0 tx=0).stop 2024-09-21T05:48:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 -- 172.21.15.28:0/1217520196 shutdown_connections 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 --2- 172.21.15.28:0/1217520196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420108d30 0x7f942010f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 --2- 172.21.15.28:0/1217520196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f94201087f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 --2- 172.21.15.28:0/1217520196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420102aa0 0x7f9420102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 -- 172.21.15.28:0/1217520196 >> 172.21.15.28:0/1217520196 conn(0x7f9420079750 msgr2=0x7f9420079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 -- 172.21.15.28:0/1217520196 shutdown_connections 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 -- 172.21.15.28:0/1217520196 wait complete. 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.344+0000 7f9426134640 1 Processor -- start 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 -- start start 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420102aa0 0x7f942019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420108d30 0x7f942019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94201129f0 con 0x7f94201083e0 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112870 con 0x7f9420108d30 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f9426134640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112b70 con 0x7f9420102aa0 2024-09-21T05:48:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42672/0 (socket says 172.21.15.28:42672) 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 -- 172.21.15.28:0/953285336 learned_addr learned my addr 172.21.15.28:0/953285336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941f7fe640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420102aa0 0x7f942019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941ffff640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420108d30 0x7f942019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 -- 172.21.15.28:0/953285336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420102aa0 msgr2=0x7f942019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420102aa0 0x7f942019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 -- 172.21.15.28:0/953285336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420108d30 msgr2=0x7f942019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420108d30 0x7f942019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.345+0000 7f941effd640 1 -- 172.21.15.28:0/953285336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94201a0420 con 0x7f94201083e0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f941effd640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9414006010 tx=0x7f9414004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f941cff9640 1 -- 172.21.15.28:0/953285336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941403f030 con 0x7f94201083e0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94201a06b0 con 0x7f94201083e0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f941cff9640 1 -- 172.21.15.28:0/953285336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9414003aa0 con 0x7f94201083e0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f941cff9640 1 -- 172.21.15.28:0/953285336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414035070 con 0x7f94201083e0 2024-09-21T05:48:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.346+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94201a6e70 con 0x7f94201083e0 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.348+0000 7f942c6a8740 1 Processor -- start 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 -- start start 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a83800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a83d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5559619f9ea0 con 0x5559616dc070 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5559619e92d0 con 0x5559616b31b0 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f942c6a8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5559619ed0e0 con 0x5559616bb390 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42684/0 (socket says 172.21.15.28:42684) 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 -- 172.21.15.28:0/2607924096 learned_addr learned my addr 172.21.15.28:0/2607924096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941cff9640 1 -- 172.21.15.28:0/953285336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9414042030 con 0x7f94201083e0 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941f7fe640 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a83800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 -- 172.21.15.28:0/2607924096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 msgr2=0x555961a83d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a83d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 -- 172.21.15.28:0/2607924096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 msgr2=0x555961a83800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a83800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941ffff640 1 -- 172.21.15.28:0/2607924096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555961747650 con 0x5559616dc070 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.349+0000 7f941f7fe640 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a83800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f941ffff640 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f94180039f0 tx=0x7f9418003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238ab40f403c6117 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f9405ffb640 1 -- 172.21.15.28:0/2607924096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941801a470 con 0x5559616dc070 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f9405ffb640 1 -- 172.21.15.28:0/2607924096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9418015070 con 0x5559616dc070 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f9405ffb640 1 -- 172.21.15.28:0/2607924096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180107c0 con 0x5559616dc070 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f941cff9640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 0x7f94080b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f941f7fe640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 0x7f94080b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.350+0000 7f941cff9640 1 -- 172.21.15.28:0/953285336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9414107b20 con 0x7f94201083e0 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 -- 172.21.15.28:0/2607924096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 msgr2=0x555961a832c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f941f7fe640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 0x7f94080b8b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x555961a81f40 tx=0x7f9410005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f94180039f0 tx=0x7f9418003eb0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 -- 172.21.15.28:0/2607924096 shutdown_connections 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a83d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a83800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 --2- 172.21.15.28:0/2607924096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a832c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 -- 172.21.15.28:0/2607924096 >> 172.21.15.28:0/2607924096 conn(0x5559616d0460 msgr2=0x555961a811b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 -- 172.21.15.28:0/2607924096 shutdown_connections 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.351+0000 7f942c6a8740 1 -- 172.21.15.28:0/2607924096 wait complete. 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 -- start start 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 Processor -- start 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a92e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a93360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55596184a700 con 0x5559616dc070 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555961984160 con 0x5559616b31b0 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555961a95e00 con 0x5559616bb390 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f942c6a8740 10 client.0 _ll_register_callbacks cb 0x55596199d9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f941ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f941ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42696/0 (socket says 172.21.15.28:42696) 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f941ffff640 1 -- 172.21.15.28:0/1136090817 learned_addr learned my addr 172.21.15.28:0/1136090817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f941f7fe640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a92e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.352+0000 7f941effd640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a93360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 -- 172.21.15.28:0/1136090817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 msgr2=0x555961a93360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a93360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 -- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 msgr2=0x555961a92e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a92e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 -- 172.21.15.28:0/1136090817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559619b1c90 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f941ffff640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f941800a010 tx=0x7f9418004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6401c415d18822 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f93feffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418011850 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f94180119f0 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9418015070 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9418031050 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f942c6a8740 10 client.16332 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.353+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555961aa1be0 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9418015070 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f94180236a0 con 0x5559616dc070 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 10 client.16332 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 10 client.16332 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 20 client.16332 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55596169a9f0 0x555961aa4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 4 client.16332 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559619fbdf0 con 0x55596169a9f0 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f941f7fe640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55596169a9f0 0x555961aa4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.354+0000 7f93feffd640 10 client.16332 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f941f7fe640 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55596169a9f0 0x555961aa4ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc1199185e7098af server_cookie=30c9cd90de875442 in_seq=0 out_seq=0 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f93feffd640 1 -- 172.21.15.28:0/1136090817 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5559619fbdf0 con 0x55596169a9f0 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f93feffd640 10 client.16332 handle_command_reply: tid=0 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f942c6a8740 1 client.16332 shutdown 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 msgr2=0x555961a9f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f941800a010 tx=0x7f9418004990 comp rx=0 tx=0).stop 2024-09-21T05:48:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 shutdown_connections 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.355+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55596169a9f0 0x555961aa4ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616dc070 0x555961a9f160 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f93feffd640 0 client.16332 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 --2- 172.21.15.28:0/1136090817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5559616b31b0 0x555961a92e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 >> 172.21.15.28:0/1136090817 conn(0x5559616d0460 msgr2=0x555961a7f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 shutdown_connections 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 1 -- 172.21.15.28:0/1136090817 wait complete. 2024-09-21T05:48:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.356+0000 7f942c6a8740 20 client.16332 trim_cache size 0 max 16384 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 Processor -- start 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 -- start start 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961a932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a90010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ad500 con 0x5559616b31b0 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94201af130 con 0x55596169a9f0 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f942c6a8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94201adbd0 con 0x5559616bb390 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f941effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f941effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42702/0 (socket says 172.21.15.28:42702) 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f941effd640 1 -- 172.21.15.28:0/3291805278 learned_addr learned my addr 172.21.15.28:0/3291805278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f941ffff640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a90010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.369+0000 7f941f7fe640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961a932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 -- 172.21.15.28:0/3291805278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 msgr2=0x555961a90010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a90010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 -- 172.21.15.28:0/3291805278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 msgr2=0x555961a932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961a932e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 -- 172.21.15.28:0/3291805278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555961a840b0 con 0x5559616b31b0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941ffff640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a90010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f941effd640 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f94140ca010 tx=0x7f9414000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae262b09df17765 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f94057fa640 1 -- 172.21.15.28:0/3291805278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414049070 con 0x5559616b31b0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f94057fa640 1 -- 172.21.15.28:0/3291805278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94140b6070 con 0x5559616b31b0 2024-09-21T05:48:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.370+0000 7f94057fa640 1 -- 172.21.15.28:0/3291805278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140c9cb0 con 0x5559616b31b0 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- 172.21.15.28:0/3291805278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 msgr2=0x555961a9f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f94140ca010 tx=0x7f9414000bc0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- 172.21.15.28:0/3291805278 shutdown_connections 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961a90010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961a9f0d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 --2- 172.21.15.28:0/3291805278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961a932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- 172.21.15.28:0/3291805278 >> 172.21.15.28:0/3291805278 conn(0x5559616d0460 msgr2=0x555961a92bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- 172.21.15.28:0/3291805278 shutdown_connections 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- 172.21.15.28:0/3291805278 wait complete. 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.371+0000 7f942c6a8740 1 -- start start 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 Processor -- start 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961aa5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961aa6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ae9a0 con 0x5559616b31b0 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ae2a0 con 0x55596169a9f0 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ab4e0 con 0x5559616bb390 2024-09-21T05:48:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f942c6a8740 10 client.0 _ll_register_callbacks cb 0x7f9418009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f941f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f941f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:49264/0 (socket says 172.21.15.28:49264) 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f941effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961aa5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f941f7fe640 1 -- 172.21.15.28:0/756818536 learned_addr learned my addr 172.21.15.28:0/756818536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.372+0000 7f941ffff640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961aa6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 -- 172.21.15.28:0/756818536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 msgr2=0x555961aa6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961aa6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 -- 172.21.15.28:0/756818536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 msgr2=0x555961aa5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961aa5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 -- 172.21.15.28:0/756818536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555961aa6ad0 con 0x55596169a9f0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941effd640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961aa5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941ffff640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961aa6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f941f7fe640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9410012cb0 tx=0x7f94100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87facf052671bd78 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f93feffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410036060 con 0x55596169a9f0 2024-09-21T05:48:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9410031480 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94140b6070 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9410031680 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f942c6a8740 10 client.25898 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555961b0f7d0 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.373+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94140b6070 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f941002a7b0 con 0x55596169a9f0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 10 client.25898 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 10 client.25898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 20 client.25898 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5559616dc070 0x555961b12020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 4 client.25898 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559619f78a0 con 0x5559616dc070 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f941effd640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5559616dc070 0x555961b12020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f93feffd640 10 client.25898 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.374+0000 7f941effd640 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5559616dc070 0x555961b12020 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e1ddf077dc4b5ce server_cookie=8968b5514d0f34e7 in_seq=0 out_seq=0 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f93feffd640 1 -- 172.21.15.28:0/756818536 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559619f78a0 con 0x5559616dc070 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f93feffd640 10 client.25898 handle_command_reply: tid=0 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 client.25898 shutdown 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 msgr2=0x555961aa88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9410012cb0 tx=0x7f94100046c0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 shutdown_connections 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5559616dc070 0x555961b12020 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5559616bb390 0x555961aa6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559616b31b0 0x555961aa5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f93feffd640 0 client.25898 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 --2- 172.21.15.28:0/756818536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55596169a9f0 0x555961aa88a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 >> 172.21.15.28:0/756818536 conn(0x5559616d0460 msgr2=0x555961a81100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 shutdown_connections 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 1 -- 172.21.15.28:0/756818536 wait complete. 2024-09-21T05:48:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.375+0000 7f942c6a8740 20 client.25898 trim_cache size 0 max 16384 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 msgr2=0x7f94080b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 0x7f94080b8b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x555961a81f40 tx=0x7f9410005f90 comp rx=0 tx=0).stop 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 msgr2=0x7f942019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9414006010 tx=0x7f9414004060 comp rx=0 tx=0).stop 2024-09-21T05:48:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 shutdown_connections 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9420108d30 0x7f942019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94080b6680 0x7f94080b8b90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94201083e0 0x7f942019aa50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 --2- 172.21.15.28:0/953285336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9420102aa0 0x7f942019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 >> 172.21.15.28:0/953285336 conn(0x7f9420079750 msgr2=0x7f94201019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 shutdown_connections 2024-09-21T05:48:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:54.376+0000 7f9426134640 1 -- 172.21.15.28:0/953285336 wait complete. 2024-09-21T05:48:54.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:54 smithi181 ceph-mon[34819]: pgmap v381: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 MiB/s wr, 7.83k op/s 2024-09-21T05:48:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:54 smithi028 ceph-mon[31227]: pgmap v381: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 MiB/s wr, 7.83k op/s 2024-09-21T05:48:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:54 smithi129 ceph-mon[34885]: pgmap v381: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 MiB/s wr, 7.83k op/s 2024-09-21T05:48:55.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:48:56.063 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 -- 172.21.15.28:0/4175032245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109430 msgr2=0x7f295c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 --2- 172.21.15.28:0/4175032245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109430 0x7f295c109840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2950009900 tx=0x7f295002a410 comp rx=0 tx=0).stop 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 -- 172.21.15.28:0/4175032245 shutdown_connections 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 --2- 172.21.15.28:0/4175032245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f295c109d80 0x7f295c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 --2- 172.21.15.28:0/4175032245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109430 0x7f295c109840 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 --2- 172.21.15.28:0/4175032245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c102060 0x7f295c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 -- 172.21.15.28:0/4175032245 >> 172.21.15.28:0/4175032245 conn(0x7f295c079e90 msgr2=0x7f295c07a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 -- 172.21.15.28:0/4175032245 shutdown_connections 2024-09-21T05:48:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 -- 172.21.15.28:0/4175032245 wait complete. 2024-09-21T05:48:56.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.067+0000 7f296172b640 1 Processor -- start 2024-09-21T05:48:56.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 -- start start 2024-09-21T05:48:56.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f295c102060 0x7f295c102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 0x7f295c103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f295c113a90 con 0x7f295c109d80 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f295c113910 con 0x7f295c109430 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f296172b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f295c113c10 con 0x7f295c102060 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57650/0 (socket says 172.21.15.28:57650) 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f295b7fe640 1 -- 172.21.15.28:0/3415817642 learned_addr learned my addr 172.21.15.28:0/3415817642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.068+0000 7f295a7fc640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 0x7f295c103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 -- 172.21.15.28:0/3415817642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f295c102060 msgr2=0x7f295c102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f295c102060 0x7f295c102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 -- 172.21.15.28:0/3415817642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 msgr2=0x7f295c103530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 0x7f295c103530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 -- 172.21.15.28:0/3415817642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295c1a8740 con 0x7f295c109d80 2024-09-21T05:48:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295a7fc640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 0x7f295c103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f295b7fe640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f295400efd0 tx=0x7f295400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f2943fff640 1 -- 172.21.15.28:0/3415817642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954003b70 con 0x7f295c109d80 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f2943fff640 1 -- 172.21.15.28:0/3415817642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2954003d10 con 0x7f295c109d80 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295c1a8a30 con 0x7f295c109d80 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f2943fff640 1 -- 172.21.15.28:0/3415817642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29540108a0 con 0x7f295c109d80 2024-09-21T05:48:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.069+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f295c1a8f00 con 0x7f295c109d80 2024-09-21T05:48:56.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.072+0000 7f296869c740 1 Processor -- start 2024-09-21T05:48:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f2943fff640 1 -- 172.21.15.28:0/3415817642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f2954010a40 con 0x7f295c109d80 2024-09-21T05:48:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 -- start start 2024-09-21T05:48:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c8f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8f4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55697c8686f0 con 0x55697c5252f0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55697c857b20 con 0x55697c54e1b0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.073+0000 7f296869c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55697c85b930 con 0x55697c52d4d0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57658/0 (socket says 172.21.15.28:57658) 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 -- 172.21.15.28:0/233904013 learned_addr learned my addr 172.21.15.28:0/233904013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295a7fc640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c8f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295a7fc640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 -- 172.21.15.28:0/233904013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 msgr2=0x55697c8f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8f4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 -- 172.21.15.28:0/233904013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 msgr2=0x55697c8f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c8f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f2943fff640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 0x7f29440b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 -- 172.21.15.28:0/233904013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55697c81d530 con 0x55697c5252f0 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295a7fc640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c8f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f2943fff640 1 -- 172.21.15.28:0/3415817642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f29540e7c20 con 0x7f295c109d80 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295a7fc640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 0x7f29440b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.074+0000 7f295affd640 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f294c009be0 tx=0x7f294c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d70040df4917ef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.075+0000 7f29417fa640 1 -- 172.21.15.28:0/233904013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c005020 con 0x55697c5252f0 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.075+0000 7f29417fa640 1 -- 172.21.15.28:0/233904013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294c0047a0 con 0x55697c5252f0 2024-09-21T05:48:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.075+0000 7f29417fa640 1 -- 172.21.15.28:0/233904013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c0186d0 con 0x55697c5252f0 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- 172.21.15.28:0/233904013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 msgr2=0x55697c8f3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f294c009be0 tx=0x7f294c00e720 comp rx=0 tx=0).stop 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f295a7fc640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 0x7f29440b8bc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f295002aa00 tx=0x7f2950005780 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- 172.21.15.28:0/233904013 shutdown_connections 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8f3ef0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 --2- 172.21.15.28:0/233904013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c8f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- 172.21.15.28:0/233904013 >> 172.21.15.28:0/233904013 conn(0x55697c542300 msgr2=0x55697c8f18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- 172.21.15.28:0/233904013 shutdown_connections 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- 172.21.15.28:0/233904013 wait complete. 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 -- start start 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.076+0000 7f296869c740 1 Processor -- start 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8ff570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c90b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55697c78fab0 con 0x55697c5252f0 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55697c7fd730 con 0x55697c54e1b0 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55697c902130 con 0x55697c52d4d0 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f296869c740 10 client.0 _ll_register_callbacks cb 0x55697c8516b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57666/0 (socket says 172.21.15.28:57666) 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295a7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8ff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295affd640 1 -- 172.21.15.28:0/1131942351 learned_addr learned my addr 172.21.15.28:0/1131942351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c90b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 -- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 msgr2=0x55697c8ff570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8ff570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 -- 172.21.15.28:0/1131942351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 msgr2=0x55697c8ff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295b7fe640 1 -- 172.21.15.28:0/1131942351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55697c5b9790 con 0x55697c54e1b0 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295a7fc640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.077+0000 7f295affd640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f295b7fe640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c90b450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f29540e5390 tx=0x7f29540d1280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a402503e17bda158 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f293eff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:48:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954009820 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f29540accf0 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c0047a0 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f29540ad5a0 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f296869c740 10 client.25910 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55697c632620 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.078+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294c0047a0 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f295401f020 con 0x55697c54e1b0 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 10 client.25910 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 10 client.25910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 20 client.25910 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c50cd80 0x55697c911340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 4 client.25910 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55697c86a640 con 0x55697c50cd80 2024-09-21T05:48:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.079+0000 7f295affd640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c50cd80 0x55697c911340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.080+0000 7f293eff5640 10 client.25910 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.080+0000 7f295affd640 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c50cd80 0x55697c911340 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bcc14fd97bc7a00 server_cookie=ae8e26dbb2fc4e30 in_seq=0 out_seq=0 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.080+0000 7f293eff5640 1 -- 172.21.15.28:0/1131942351 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55697c86a640 con 0x55697c50cd80 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.080+0000 7f293eff5640 10 client.25910 handle_command_reply: tid=0 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.080+0000 7f296869c740 1 client.25910 shutdown 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 msgr2=0x55697c90b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c90b450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f29540e5390 tx=0x7f29540d1280 comp rx=0 tx=0).stop 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 shutdown_connections 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c50cd80 0x55697c911340 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f293eff5640 0 client.25910 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c54e1b0 0x55697c90b450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c8ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 --2- 172.21.15.28:0/1131942351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c5252f0 0x55697c8ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 >> 172.21.15.28:0/1131942351 conn(0x55697c542300 msgr2=0x55697c8f49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:56.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 shutdown_connections 2024-09-21T05:48:56.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 1 -- 172.21.15.28:0/1131942351 wait complete. 2024-09-21T05:48:56.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.081+0000 7f296869c740 20 client.25910 trim_cache size 0 max 16384 2024-09-21T05:48:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.098+0000 7f296869c740 1 Processor -- start 2024-09-21T05:48:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.098+0000 7f296869c740 1 -- start start 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c8f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c90b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1b1900 con 0x55697c50cd80 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1b3530 con 0x55697c5252f0 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f296869c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1b1fd0 con 0x55697c52d4d0 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c8f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c90b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34628/0 (socket says 172.21.15.28:34628) 2024-09-21T05:48:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 -- 172.21.15.28:0/2791570815 learned_addr learned my addr 172.21.15.28:0/2791570815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 -- 172.21.15.28:0/2791570815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 msgr2=0x55697c90b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c90b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 -- 172.21.15.28:0/2791570815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 msgr2=0x55697c8f50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c8f50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 -- 172.21.15.28:0/2791570815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55697c8fd890 con 0x55697c5252f0 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295affd640 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c8f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.099+0000 7f295a7fc640 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2950031440 tx=0x7f2950009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f128f58c2b8167f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f2940ff9640 1 -- 172.21.15.28:0/2791570815 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950031620 con 0x55697c5252f0 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f2940ff9640 1 -- 172.21.15.28:0/2791570815 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2950031c70 con 0x55697c5252f0 2024-09-21T05:48:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f2940ff9640 1 -- 172.21.15.28:0/2791570815 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950056bf0 con 0x55697c5252f0 2024-09-21T05:48:56.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 -- 172.21.15.28:0/2791570815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 msgr2=0x55697c90b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:56.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2950031440 tx=0x7f2950009260 comp rx=0 tx=0).stop 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 -- 172.21.15.28:0/2791570815 shutdown_connections 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c52d4d0 0x55697c90b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c5252f0 0x55697c90b350 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 --2- 172.21.15.28:0/2791570815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c8f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.100+0000 7f296869c740 1 -- 172.21.15.28:0/2791570815 >> 172.21.15.28:0/2791570815 conn(0x55697c542300 msgr2=0x55697c8f49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- 172.21.15.28:0/2791570815 shutdown_connections 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- 172.21.15.28:0/2791570815 wait complete. 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- start start 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 Processor -- start 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 0x55697c9050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c52d4d0 0x55697c979d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1b2da0 con 0x55697c50cd80 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1b26a0 con 0x55697c52d4d0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f295c1af8e0 con 0x55697c5252f0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f296869c740 10 client.0 _ll_register_callbacks cb 0x7f2954003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f295affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57688/0 (socket says 172.21.15.28:57688) 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f295a7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 0x55697c9050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f295affd640 1 -- 172.21.15.28:0/1785655264 learned_addr learned my addr 172.21.15.28:0/1785655264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.101+0000 7f295b7fe640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c52d4d0 0x55697c979d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 -- 172.21.15.28:0/1785655264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c52d4d0 msgr2=0x55697c979d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c52d4d0 0x55697c979d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 -- 172.21.15.28:0/1785655264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 msgr2=0x55697c904b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 -- 172.21.15.28:0/1785655264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55697c97a310 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295affd640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f295a7fc640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 0x55697c9050f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2950009230 tx=0x7f2950056690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a954a07d40991603 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f293eff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950031830 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2950031c70 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f29500319d0 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2950063050 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f296869c740 10 client.25954 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.102+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55697c97b5a0 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2950031c70 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f295005d020 con 0x55697c5252f0 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 10 client.25954 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 10 client.25954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 20 client.25954 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c54e1b0 0x55697c97ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 4 client.25954 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55697c8660f0 con 0x55697c54e1b0 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f295affd640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c54e1b0 0x55697c97ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f293eff5640 10 client.25954 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:48:56.103+0000 7f295affd640 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c54e1b0 0x55697c97ddb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38980bd0aa001ac4 server_cookie=5646b58c1f587d90 in_seq=0 out_seq=0 2024-09-21T05:48:56.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:56 smithi028 ceph-mon[31227]: pgmap v382: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 47 MiB/s wr, 5.97k op/s 2024-09-21T05:48:56.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:56 smithi181 ceph-mon[34819]: pgmap v382: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 47 MiB/s wr, 5.97k op/s 2024-09-21T05:48:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:56 smithi129 ceph-mon[34885]: pgmap v382: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 47 MiB/s wr, 5.97k op/s 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 -- 172.21.15.28:0/301276037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c108cf0 msgr2=0x7f5b8c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/301276037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c10f520 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009ca0 tx=0x7f5b8402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 -- 172.21.15.28:0/301276037 shutdown_connections 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/301276037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c10f520 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/301276037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/301276037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 -- 172.21.15.28:0/301276037 >> 172.21.15.28:0/301276037 conn(0x7f5b8c079730 msgr2=0x7f5b8c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 -- 172.21.15.28:0/301276037 shutdown_connections 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.044+0000 7f5b92f9e640 1 -- 172.21.15.28:0/301276037 wait complete. 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 Processor -- start 2024-09-21T05:48:57.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 -- start start 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113a80 con 0x7f5b8c1083a0 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113900 con 0x7f5b8c102a60 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b92f9e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113c00 con 0x7f5b8c108cf0 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57710/0 (socket says 172.21.15.28:57710) 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b8bfff640 1 -- 172.21.15.28:0/920311092 learned_addr learned my addr 172.21.15.28:0/920311092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.045+0000 7f5b9153b640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 -- 172.21.15.28:0/920311092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c108cf0 msgr2=0x7f5b8c19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 -- 172.21.15.28:0/920311092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 msgr2=0x7f5b8c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 -- 172.21.15.28:0/920311092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c1a0380 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9c0 tx=0x7f5b7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b89ffb640 1 -- 172.21.15.28:0/920311092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7800cd80 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b89ffb640 1 -- 172.21.15.28:0/920311092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b780054d0 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c1a0670 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b89ffb640 1 -- 172.21.15.28:0/920311092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78010430 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.046+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c1a6dd0 con 0x7f5b8c1083a0 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 Processor -- start 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 -- start start 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f521b0 0x561966322be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f5a390 0x561966323120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561966298ea0 con 0x561965f521b0 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.049+0000 7f5b99512740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619662882d0 con 0x561965f7b070 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b99512740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56196628c0e0 con 0x561965f5a390 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b90d3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f521b0 0x561966322be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34644/0 (socket says 172.21.15.28:34644) 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 -- 172.21.15.28:0/4112855282 learned_addr learned my addr 172.21.15.28:0/4112855282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 -- 172.21.15.28:0/4112855282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f5a390 msgr2=0x561966323120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b89ffb640 1 -- 172.21.15.28:0/920311092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5b780105d0 con 0x7f5b8c1083a0 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f5a390 0x561966323120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 -- 172.21.15.28:0/4112855282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f521b0 msgr2=0x561966322be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f521b0 0x561966322be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 -- 172.21.15.28:0/4112855282 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561965fe6650 con 0x561965f7b070 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b9153b640 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b8402f410 tx=0x7f5b8402f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e63d1b9d52cf10af server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.050+0000 7f5b76ffd640 1 -- 172.21.15.28:0/4112855282 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84031cf0 con 0x561965f7b070 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.051+0000 7f5b76ffd640 1 -- 172.21.15.28:0/4112855282 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b84036030 con 0x561965f7b070 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.051+0000 7f5b76ffd640 1 -- 172.21.15.28:0/4112855282 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b840042a0 con 0x561965f7b070 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.051+0000 7f5b89ffb640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 0x7f5b7c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.051+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 0x7f5b7c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.051+0000 7f5b89ffb640 1 -- 172.21.15.28:0/920311092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5b780e7af0 con 0x7f5b8c1083a0 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- 172.21.15.28:0/4112855282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 msgr2=0x5619663226a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 0x7f5b7c0b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x561966321320 tx=0x7f5b80015040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b8402f410 tx=0x7f5b8402f440 comp rx=0 tx=0).stop 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- 172.21.15.28:0/4112855282 shutdown_connections 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f5a390 0x561966323120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f521b0 0x561966322be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 --2- 172.21.15.28:0/4112855282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x5619663226a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- 172.21.15.28:0/4112855282 >> 172.21.15.28:0/4112855282 conn(0x561965f6f460 msgr2=0x561966320590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- 172.21.15.28:0/4112855282 shutdown_connections 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- 172.21.15.28:0/4112855282 wait complete. 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 -- start start 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.052+0000 7f5b99512740 1 Processor -- start 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x56196632e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619660e9700 con 0x561965f7b070 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561966223160 con 0x561965f5a390 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561966330ca0 con 0x561965f521b0 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b99512740 10 client.0 _ll_register_callbacks cb 0x56196623c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57720/0 (socket says 172.21.15.28:57720) 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 -- 172.21.15.28:0/923182567 learned_addr learned my addr 172.21.15.28:0/923182567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x56196632e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 -- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 msgr2=0x56196632ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 -- 172.21.15.28:0/923182567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 msgr2=0x56196632e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x56196632e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.053+0000 7f5b9153b640 1 -- 172.21.15.28:0/923182567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561966250c90 con 0x561965f7b070 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b9153b640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005e30 tx=0x7f5b84003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4243ab97c85f6adc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8404a040 con 0x561965f7b070 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b84036030 con 0x561965f7b070 2024-09-21T05:48:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5b84031650 con 0x561965f7b070 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b84031850 con 0x561965f7b070 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b99512740 10 client.16368 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56196633c9a0 con 0x561965f7b070 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.054+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b84036030 con 0x561965f7b070 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f5b8404d030 con 0x561965f7b070 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 10 client.16368 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 20 client.16368 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f399f0 0x56196633fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 4 client.16368 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56196629adf0 con 0x561965f399f0 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f399f0 0x56196633fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f399f0 0x56196633fd00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1660d44004c64d0 server_cookie=59b87c98d717b1db in_seq=0 out_seq=0 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.055+0000 7f5b6bfff640 10 client.16368 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b6bfff640 1 -- 172.21.15.28:0/923182567 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56196629adf0 con 0x561965f399f0 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b6bfff640 10 client.16368 handle_command_reply: tid=0 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 client.16368 shutdown 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 msgr2=0x56196632e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005e30 tx=0x7f5b84003c40 comp rx=0 tx=0).stop 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 shutdown_connections 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f399f0 0x56196633fd00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x56196632e880 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x56196632e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b6bfff640 0 client.16368 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 --2- 172.21.15.28:0/923182567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 >> 172.21.15.28:0/923182567 conn(0x561965f6f460 msgr2=0x561966325830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 shutdown_connections 2024-09-21T05:48:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.056+0000 7f5b99512740 1 -- 172.21.15.28:0/923182567 wait complete. 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.057+0000 7f5b99512740 20 client.16368 trim_cache size 0 max 16384 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.070+0000 7f5b99512740 1 Processor -- start 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.070+0000 7f5b99512740 1 -- start start 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x56196632b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ad3b0 con 0x561965f5a390 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1aefe0 con 0x561965f7b070 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b99512740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ada80 con 0x561965f521b0 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57730/0 (socket says 172.21.15.28:57730) 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 -- 172.21.15.28:0/868438825 learned_addr learned my addr 172.21.15.28:0/868438825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b9153b640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x56196632b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 -- 172.21.15.28:0/868438825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 msgr2=0x56196632f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 -- 172.21.15.28:0/868438825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 msgr2=0x56196632b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x56196632b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 -- 172.21.15.28:0/868438825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561966324bc0 con 0x561965f5a390 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b9153b640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x56196632b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.071+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800cfd0 tx=0x7f5b780ad7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6ba5160a9a5904 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.072+0000 7f5b767fc640 1 -- 172.21.15.28:0/868438825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780a90d0 con 0x561965f5a390 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.072+0000 7f5b767fc640 1 -- 172.21.15.28:0/868438825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b780046c0 con 0x561965f5a390 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.072+0000 7f5b767fc640 1 -- 172.21.15.28:0/868438825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780b7380 con 0x561965f5a390 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- 172.21.15.28:0/868438825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 msgr2=0x56196632afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800cfd0 tx=0x7f5b780ad7f0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- 172.21.15.28:0/868438825 shutdown_connections 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f7b070 0x56196632b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f5a390 0x56196632afb0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 --2- 172.21.15.28:0/868438825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x56196632f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- 172.21.15.28:0/868438825 >> 172.21.15.28:0/868438825 conn(0x561965f6f460 msgr2=0x56196632f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- 172.21.15.28:0/868438825 shutdown_connections 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- 172.21.15.28:0/868438825 wait complete. 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 -- start start 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.073+0000 7f5b99512740 1 Processor -- start 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x561966332df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ae850 con 0x561965f7b070 2024-09-21T05:48:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ae150 con 0x561965f5a390 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1ab390 con 0x561965f521b0 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b99512740 10 client.0 _ll_register_callbacks cb 0x7f5b80009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b90d3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57734/0 (socket says 172.21.15.28:57734) 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b90d3a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33254/0 (socket says 172.21.15.28:33254) 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 -- 172.21.15.28:0/1368310318 learned_addr learned my addr 172.21.15.28:0/1368310318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b8bfff640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x561966332df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 -- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 msgr2=0x561966332810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 -- 172.21.15.28:0/1368310318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 msgr2=0x561966332df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x561966332df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b9153b640 1 -- 172.21.15.28:0/1368310318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619663a3c40 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.074+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b9153b640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5b84006010 tx=0x7f5b84009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eef3e1f3e1dbb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84031450 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5b840315f0 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56196631b9e0 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b8402f660 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 10 client.16374 fetch_fsmap learned FSMap version 31 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5619663aaf60 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56196631b9e0 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7f5b84069070 con 0x561965f7b070 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 10 client.16374 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 20 client.16374 populate_metadata read hostname 'smithi028' 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f0eda0 0x5619663ad780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:48:57.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 4 client.16374 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.075+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5619662968a0 con 0x561965f0eda0 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f0eda0 0x5619663ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b6bfff640 10 client.16374 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b90d3a640 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f0eda0 0x5619663ad780 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adff4bfd6fb16f72 server_cookie=f7b1dfba9eeaeb9e in_seq=0 out_seq=0 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b6bfff640 1 -- 172.21.15.28:0/1368310318 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5619662968a0 con 0x561965f0eda0 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b6bfff640 10 client.16374 handle_command_reply: tid=0 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 client.16374 shutdown 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:48:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 msgr2=0x5619663a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5b84006010 tx=0x7f5b84009b20 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 shutdown_connections 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561965f0eda0 0x5619663ad780 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561965f7b070 0x5619663a35d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561965f5a390 0x561966332df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b6bfff640 0 client.16374 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 --2- 172.21.15.28:0/1368310318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561965f521b0 0x561966332810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.076+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 >> 172.21.15.28:0/1368310318 conn(0x561965f6f460 msgr2=0x56196632c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 shutdown_connections 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b99512740 1 -- 172.21.15.28:0/1368310318 wait complete. 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b99512740 20 client.16374 trim_cache size 0 max 16384 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 msgr2=0x7f5b7c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 0x7f5b7c0b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x561966321320 tx=0x7f5b80015040 comp rx=0 tx=0).stop 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 msgr2=0x7f5b8c19a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:48:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9c0 tx=0x7f5b7800ee80 comp rx=0 tx=0).stop 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 shutdown_connections 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b8c108cf0 0x7f5b8c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5b7c0b6680 0x7f5b7c0b8b90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b8c1083a0 0x7f5b8c19a9b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 --2- 172.21.15.28:0/920311092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b8c102a60 0x7f5b8c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 >> 172.21.15.28:0/920311092 conn(0x7f5b8c079730 msgr2=0x7f5b8c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.077+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 shutdown_connections 2024-09-21T05:48:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:48:57.078+0000 7f5b92f9e640 1 -- 172.21.15.28:0/920311092 wait complete. 2024-09-21T05:48:58.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:48:58 smithi028 ceph-mon[31227]: pgmap v383: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 54 MiB/s wr, 6.85k op/s 2024-09-21T05:48:58.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:48:58 smithi181 ceph-mon[34819]: pgmap v383: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 54 MiB/s wr, 6.85k op/s 2024-09-21T05:48:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:48:58 smithi129 ceph-mon[34885]: pgmap v383: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 54 MiB/s wr, 6.85k op/s 2024-09-21T05:48:59.515 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:48:59.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:48:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:00 smithi181 ceph-mon[34819]: pgmap v384: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 49 MiB/s wr, 6.13k op/s 2024-09-21T05:49:00.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:00 smithi028 ceph-mon[31227]: pgmap v384: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 49 MiB/s wr, 6.13k op/s 2024-09-21T05:49:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.597+0000 7fcaf53a0640 1 -- 172.21.15.28:0/330296665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 msgr2=0x7fcaf010a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.597+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/330296665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 0x7fcaf010a3c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009860 tx=0x7fcae002a360 comp rx=0 tx=0).stop 2024-09-21T05:49:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.599+0000 7fcaf53a0640 1 -- 172.21.15.28:0/330296665 shutdown_connections 2024-09-21T05:49:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.599+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/330296665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf011f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.599+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/330296665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf010adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.599+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/330296665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 0x7fcaf010a3c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.599+0000 7fcaf53a0640 1 -- 172.21.15.28:0/330296665 >> 172.21.15.28:0/330296665 conn(0x7fcaf0070ec0 msgr2=0x7fcaf00712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.602+0000 7fcaf53a0640 1 -- 172.21.15.28:0/330296665 shutdown_connections 2024-09-21T05:49:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.602+0000 7fcaf53a0640 1 -- 172.21.15.28:0/330296665 wait complete. 2024-09-21T05:49:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.603+0000 7fcaf53a0640 1 Processor -- start 2024-09-21T05:49:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.604+0000 7fcaf53a0640 1 -- start start 2024-09-21T05:49:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.605+0000 7fcaf53a0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf011c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.605+0000 7fcaf53a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 0x7fcaf0118720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.605+0000 7fcaf53a0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.606+0000 7fcaf53a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0114c10 con 0x7fcaf0111f40 2024-09-21T05:49:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.606+0000 7fcaf53a0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0114a90 con 0x7fcaf011cc30 2024-09-21T05:49:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.606+0000 7fcaf53a0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0114d90 con 0x7fcaf010a9e0 2024-09-21T05:49:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.607+0000 7fcaef7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.607+0000 7fcaef7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34684/0 (socket says 172.21.15.28:34684) 2024-09-21T05:49:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.607+0000 7fcaef7fe640 1 -- 172.21.15.28:0/3311203546 learned_addr learned my addr 172.21.15.28:0/3311203546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.607+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf011c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaef7fe640 1 -- 172.21.15.28:0/3311203546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 msgr2=0x7fcaf011c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf011c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaef7fe640 1 -- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 msgr2=0x7fcaf0118720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:49:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 0x7fcaf0118720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaef7fe640 1 -- 172.21.15.28:0/3311203546 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf01194e0 con 0x7fcaf011cc30 2024-09-21T05:49:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.608+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf011c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:49:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.609+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcae801fd90 tx=0x7fcae801d750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.609+0000 7fcad7fff640 1 -- 172.21.15.28:0/3311203546 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae801de60 con 0x7fcaf011cc30 2024-09-21T05:49:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.609+0000 7fcad7fff640 1 -- 172.21.15.28:0/3311203546 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcae8003970 con 0x7fcaf011cc30 2024-09-21T05:49:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.609+0000 7fcad7fff640 1 -- 172.21.15.28:0/3311203546 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae80043c0 con 0x7fcaf011cc30 2024-09-21T05:49:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.610+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf01ace30 con 0x7fcaf011cc30 2024-09-21T05:49:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.610+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf01ad290 con 0x7fcaf011cc30 2024-09-21T05:49:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:00 smithi129 ceph-mon[34885]: pgmap v384: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 49 MiB/s wr, 6.13k op/s 2024-09-21T05:49:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.615+0000 7fcafb914740 1 Processor -- start 2024-09-21T05:49:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.615+0000 7fcafb914740 1 -- start start 2024-09-21T05:49:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.616+0000 7fcafb914740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcafb914740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55703831a490 0x5570386ea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcafb914740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x5570386eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcafb914740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557038660e60 con 0x55703831a490 2024-09-21T05:49:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcafb914740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557038650290 con 0x557038343350 2024-09-21T05:49:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcafb914740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5570386540a0 con 0x557038322670 2024-09-21T05:49:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34696/0 (socket says 172.21.15.28:34696) 2024-09-21T05:49:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.617+0000 7fcaee7fc640 1 -- 172.21.15.28:0/3288461772 learned_addr learned my addr 172.21.15.28:0/3288461772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x5570386eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 -- 172.21.15.28:0/3288461772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 msgr2=0x5570386eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x5570386eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 -- 172.21.15.28:0/3288461772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55703831a490 msgr2=0x5570386ea970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55703831a490 0x5570386ea970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 -- 172.21.15.28:0/3288461772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570383ae930 con 0x557038343350 2024-09-21T05:49:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.618+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcae400bc90 tx=0x7fcae400b440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=667021e0fdc19f5c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.620+0000 7fcad7fff640 1 -- 172.21.15.28:0/3311203546 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fcae8004560 con 0x7fcaf011cc30 2024-09-21T05:49:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.621+0000 7fcad57fa640 1 -- 172.21.15.28:0/3288461772 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4016030 con 0x557038343350 2024-09-21T05:49:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.621+0000 7fcad57fa640 1 -- 172.21.15.28:0/3288461772 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcae400be40 con 0x557038343350 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.622+0000 7fcafb914740 1 -- 172.21.15.28:0/3288461772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 msgr2=0x5570386ea430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.622+0000 7fcafb914740 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcae400bc90 tx=0x7fcae400b440 comp rx=0 tx=0).stop 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcad7fff640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 0x7fcad80b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcad7fff640 1 -- 172.21.15.28:0/3311203546 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcae802c070 con 0x7fcaf011cc30 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 0x7fcad80b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcafb914740 1 -- 172.21.15.28:0/3288461772 shutdown_connections 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcafb914740 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x5570386eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcafb914740 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55703831a490 0x5570386ea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcafb914740 1 --2- 172.21.15.28:0/3288461772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038343350 0x5570386ea430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.623+0000 7fcafb914740 1 -- 172.21.15.28:0/3288461772 >> 172.21.15.28:0/3288461772 conn(0x557038337740 msgr2=0x5570386e8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.624+0000 7fcafb914740 1 -- 172.21.15.28:0/3288461772 shutdown_connections 2024-09-21T05:49:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.624+0000 7fcafb914740 1 -- 172.21.15.28:0/3288461772 wait complete. 2024-09-21T05:49:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.624+0000 7fcafb914740 1 -- start start 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.624+0000 7fcafb914740 1 Processor -- start 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcafb914740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 0x5570386f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcafb914740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcafb914740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x557038701fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34710/0 (socket says 172.21.15.28:34710) 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcaeeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 0x5570386f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcaeeffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 0x5570386f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33296/0 (socket says 172.21.15.28:33296) 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.625+0000 7fcaee7fc640 1 -- 172.21.15.28:0/2629546411 learned_addr learned my addr 172.21.15.28:0/2629546411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:00.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.626+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570384b1890 con 0x557038343350 2024-09-21T05:49:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.626+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570385ead40 con 0x557038322670 2024-09-21T05:49:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.626+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5570386f8c60 con 0x55703831a490 2024-09-21T05:49:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.626+0000 7fcafb914740 10 client.0 _ll_register_callbacks cb 0x55703864ec80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.627+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x557038701fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 -- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 msgr2=0x5570386f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 0x5570386f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 -- 172.21.15.28:0/2629546411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 msgr2=0x557038701fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x557038701fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570387024e0 con 0x557038322670 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x557038701fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.629+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcae4012d20 tx=0x7fcae400b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4839387c28731886 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.630+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 0x7fcad80b8d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcae002a950 tx=0x7fcae0004040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.630+0000 7fcad2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:49:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.630+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4016050 con 0x557038322670 2024-09-21T05:49:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.630+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557038662db0 con 0x557038322670 2024-09-21T05:49:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.630+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcae4011a20 con 0x557038322670 2024-09-21T05:49:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.631+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcae4011c20 con 0x557038322670 2024-09-21T05:49:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.631+0000 7fcafb914740 10 client.25946 fetch_fsmap learned FSMap version 31 2024-09-21T05:49:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.631+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557038704db0 con 0x557038322670 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.636+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcae4017040 con 0x557038322670 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.636+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fcae402c020 con 0x557038322670 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.636+0000 7fcafb914740 10 client.25946 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.636+0000 7fcafb914740 10 client.25946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.636+0000 7fcafb914740 20 client.25946 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.637+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557038301cd0 0x5570387086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.637+0000 7fcafb914740 4 client.25946 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:49:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.637+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557038662db0 con 0x557038301cd0 2024-09-21T05:49:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.639+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557038301cd0 0x5570387086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.643+0000 7fcad2ff5640 10 client.25946 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.643+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557038301cd0 0x5570387086e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8e4b07678a4a6d server_cookie=a584805525fb4e8a in_seq=0 out_seq=0 2024-09-21T05:49:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.644+0000 7fcad2ff5640 1 -- 172.21.15.28:0/2629546411 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557038662db0 con 0x557038301cd0 2024-09-21T05:49:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.644+0000 7fcad2ff5640 10 client.25946 handle_command_reply: tid=0 2024-09-21T05:49:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.644+0000 7fcafb914740 1 client.25946 shutdown 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 msgr2=0x5570386f6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcae4012d20 tx=0x7fcae400b4b0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 shutdown_connections 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557038301cd0 0x5570387086e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x557038701fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557038322670 0x5570386f6490 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 --2- 172.21.15.28:0/2629546411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703831a490 0x5570386f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 >> 172.21.15.28:0/2629546411 conn(0x557038337740 msgr2=0x5570386ed610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.645+0000 7fcad2ff5640 0 client.25946 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.646+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 shutdown_connections 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.646+0000 7fcafb914740 1 -- 172.21.15.28:0/2629546411 wait complete. 2024-09-21T05:49:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.646+0000 7fcafb914740 20 client.25946 trim_cache size 0 max 16384 2024-09-21T05:49:00.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.688+0000 7fcafb914740 1 Processor -- start 2024-09-21T05:49:00.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.690+0000 7fcafb914740 1 -- start start 2024-09-21T05:49:00.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386f0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x5570386f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b5db0 con 0x557038343350 2024-09-21T05:49:00.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b79e0 con 0x55703831a490 2024-09-21T05:49:00.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.691+0000 7fcafb914740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b6480 con 0x557038322670 2024-09-21T05:49:00.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.693+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.693+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33310/0 (socket says 172.21.15.28:33310) 2024-09-21T05:49:00.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.693+0000 7fcaee7fc640 1 -- 172.21.15.28:0/386330044 learned_addr learned my addr 172.21.15.28:0/386330044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x5570386f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaee7fc640 1 -- 172.21.15.28:0/386330044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 msgr2=0x5570386f0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386f0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaee7fc640 1 -- 172.21.15.28:0/386330044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 msgr2=0x5570386f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x5570386f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaee7fc640 1 -- 172.21.15.28:0/386330044 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570386ec9f0 con 0x557038322670 2024-09-21T05:49:00.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.694+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x5570386f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:49:00.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.696+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcae40048f0 tx=0x7fcae400b440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1193d0ec38a37be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.697+0000 7fcad4ff9640 1 -- 172.21.15.28:0/386330044 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4033030 con 0x557038322670 2024-09-21T05:49:00.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.697+0000 7fcad4ff9640 1 -- 172.21.15.28:0/386330044 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcae4004e10 con 0x557038322670 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.697+0000 7fcad4ff9640 1 -- 172.21.15.28:0/386330044 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae4016070 con 0x557038322670 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 -- 172.21.15.28:0/386330044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 msgr2=0x557038701ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcae40048f0 tx=0x7fcae400b440 comp rx=0 tx=0).stop 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 -- 172.21.15.28:0/386330044 shutdown_connections 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038343350 0x5570386f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038322670 0x557038701ea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 --2- 172.21.15.28:0/386330044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386f0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 -- 172.21.15.28:0/386330044 >> 172.21.15.28:0/386330044 conn(0x557038337740 msgr2=0x5570386f0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.699+0000 7fcafb914740 1 -- 172.21.15.28:0/386330044 shutdown_connections 2024-09-21T05:49:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.700+0000 7fcafb914740 1 -- 172.21.15.28:0/386330044 wait complete. 2024-09-21T05:49:00.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.700+0000 7fcafb914740 1 -- start start 2024-09-21T05:49:00.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.701+0000 7fcafb914740 1 Processor -- start 2024-09-21T05:49:00.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcafb914740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcafb914740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038322670 0x5570386f98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcaeeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcaeeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34722/0 (socket says 172.21.15.28:34722) 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1744745556 learned_addr learned my addr 172.21.15.28:0/1744745556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.702+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038343350 0x5570386f9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.703+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b7250 con 0x557038322670 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.703+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b6b50 con 0x55703831a490 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.703+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038343350 0x5570386f9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.703+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf01b3d90 con 0x557038343350 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.703+0000 7fcafb914740 10 client.0 _ll_register_callbacks cb 0x7fcae400baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038343350 msgr2=0x5570386f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038343350 0x5570386f9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1744745556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038322670 msgr2=0x5570386f98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038322670 0x5570386f98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570386fa370 con 0x55703831a490 2024-09-21T05:49:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.704+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcae4004e10 con 0x55703831a490 2024-09-21T05:49:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.705+0000 7fcaeeffd640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcae0004120 tx=0x7fcae002a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f571d8a837166b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.705+0000 7fcad2ff5640 10 client.25952 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:49:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.705+0000 7fcad2ff5640 1 -- 172.21.15.28:0/1744745556 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0037040 con 0x55703831a490 2024-09-21T05:49:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.706+0000 7fcad2ff5640 1 -- 172.21.15.28:0/1744745556 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcae0005240 con 0x55703831a490 2024-09-21T05:49:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.706+0000 7fcad2ff5640 1 -- 172.21.15.28:0/1744745556 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcae0005460 con 0x55703831a490 2024-09-21T05:49:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.706+0000 7fcafb914740 10 client.25952 fetch_fsmap learned FSMap version 31 2024-09-21T05:49:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.706+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=31}) -- 0x557038772cb0 con 0x55703831a490 2024-09-21T05:49:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcad2ff5640 1 -- 172.21.15.28:0/1744745556 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fcae0058070 con 0x55703831a490 2024-09-21T05:49:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 10 client.25952 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 10 client.25952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 20 client.25952 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5570382d7080 0x557038775440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 4 client.25952 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.707+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55703865e860 con 0x5570382d7080 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.708+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5570382d7080 0x557038775440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.708+0000 7fcaee7fc640 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5570382d7080 0x557038775440 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00b025a8565bf6d server_cookie=303c7aa5643ed91e in_seq=0 out_seq=0 2024-09-21T05:49:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.708+0000 7fcad2ff5640 10 client.25952 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.709+0000 7fcad2ff5640 1 -- 172.21.15.28:0/1744745556 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55703865e860 con 0x5570382d7080 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcad2ff5640 10 client.25952 handle_command_reply: tid=0 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcafb914740 1 client.25952 shutdown 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 msgr2=0x5570386fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcae0004120 tx=0x7fcae002a360 comp rx=0 tx=0).stop 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 shutdown_connections 2024-09-21T05:49:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.710+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5570382d7080 0x557038775440 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557038343350 0x5570386f9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557038322670 0x5570386f98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 --2- 172.21.15.28:0/1744745556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55703831a490 0x5570386fe3f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 >> 172.21.15.28:0/1744745556 conn(0x557038337740 msgr2=0x5570386f7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcad2ff5640 0 client.25952 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 shutdown_connections 2024-09-21T05:49:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 1 -- 172.21.15.28:0/1744745556 wait complete. 2024-09-21T05:49:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.711+0000 7fcafb914740 20 client.25952 trim_cache size 0 max 16384 2024-09-21T05:49:00.716 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:49:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:49:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "45ebe5b7-9e03-4b06-b4b1-cee561706b06", 2024-09-21T05:49:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:49:00.717 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:49:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 msgr2=0x7fcad80b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 0x7fcad80b8d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcae002a950 tx=0x7fcae0004040 comp rx=0 tx=0).stop 2024-09-21T05:49:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 msgr2=0x7fcaf0118cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcae801fd90 tx=0x7fcae801d750 comp rx=0 tx=0).stop 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 shutdown_connections 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaf011cc30 0x7fcaf0118cc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcad80b6800 0x7fcad80b8d10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0111f40 0x7fcaf0118720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 --2- 172.21.15.28:0/3311203546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaf010a9e0 0x7fcaf011c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 >> 172.21.15.28:0/3311203546 conn(0x7fcaf0070ec0 msgr2=0x7fcaf010ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 shutdown_connections 2024-09-21T05:49:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:00.715+0000 7fcaf53a0640 1 -- 172.21.15.28:0/3311203546 wait complete. 2024-09-21T05:49:01.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:01 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:49:01.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:01 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:01.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:01 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:49:01.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:01 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:49:01.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:01 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:01.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:01 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:49:01.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:01 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:49:01.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:01 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:01.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:01 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:49:02.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:49:02.387 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:49:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:02 smithi181 ceph-mon[34819]: pgmap v385: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 42 MiB/s wr, 5.20k op/s 2024-09-21T05:49:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:02 smithi028 ceph-mon[31227]: pgmap v385: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 42 MiB/s wr, 5.20k op/s 2024-09-21T05:49:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:02 smithi129 ceph-mon[34885]: pgmap v385: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 42 MiB/s wr, 5.20k op/s 2024-09-21T05:49:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 -- 172.21.15.28:0/1717540373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c1021f0 msgr2=0x7fdf9c102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/1717540373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c102600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800a970 tx=0x7fdf9802b690 comp rx=0 tx=0).stop 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 -- 172.21.15.28:0/1717540373 shutdown_connections 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/1717540373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 0x7fdf9c10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/1717540373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c107b30 0x7fdf9c107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/1717540373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c102600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 -- 172.21.15.28:0/1717540373 >> 172.21.15.28:0/1717540373 conn(0x7fdf9c077c60 msgr2=0x7fdf9c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 -- 172.21.15.28:0/1717540373 shutdown_connections 2024-09-21T05:49:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.159+0000 7fdfa43e0640 1 -- 172.21.15.28:0/1717540373 wait complete. 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.160+0000 7fdfa43e0640 1 Processor -- start 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.160+0000 7fdfa43e0640 1 -- start start 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.160+0000 7fdfa43e0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.160+0000 7fdfa43e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.160+0000 7fdfa43e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 0x7fdf9c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa43e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111900 con 0x7fdf9c108480 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa43e0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111780 con 0x7fdf9c107b30 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa43e0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111a80 con 0x7fdf9c1021f0 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34738/0 (socket says 172.21.15.28:34738) 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 -- 172.21.15.28:0/789392073 learned_addr learned my addr 172.21.15.28:0/789392073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa297d640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 0x7fdf9c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 -- 172.21.15.28:0/789392073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c1021f0 msgr2=0x7fdf9c19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 -- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 msgr2=0x7fdf9c19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 0x7fdf9c19fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 -- 172.21.15.28:0/789392073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c1a0520 con 0x7fdf9c107b30 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdfa197b640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006fd0 tx=0x7fdf8c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.161+0000 7fdf937fe640 1 -- 172.21.15.28:0/789392073 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c011830 con 0x7fdf9c107b30 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.162+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c1a0810 con 0x7fdf9c107b30 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.162+0000 7fdf937fe640 1 -- 172.21.15.28:0/789392073 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8c016070 con 0x7fdf9c107b30 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.162+0000 7fdf937fe640 1 -- 172.21.15.28:0/789392073 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c010e00 con 0x7fdf9c107b30 2024-09-21T05:49:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.162+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1a6f70 con 0x7fdf9c107b30 2024-09-21T05:49:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.164+0000 7fdfaa954740 1 Processor -- start 2024-09-21T05:49:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.164+0000 7fdfaa954740 1 -- start start 2024-09-21T05:49:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdf937fe640 1 -- 172.21.15.28:0/789392073 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fdf8c01b460 con 0x7fdf9c107b30 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e95b0070 0x5646e9957650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 0x5646e99580d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5646e98cde50 con 0x5646e95871b0 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5646e98bd280 con 0x5646e95b0070 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfaa954740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5646e98c1090 con 0x5646e958f390 2024-09-21T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57798/0 (socket says 172.21.15.28:57798) 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 -- 172.21.15.28:0/2428803536 learned_addr learned my addr 172.21.15.28:0/2428803536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa297d640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e95b0070 0x5646e9957650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa197b640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 0x5646e99580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 -- 172.21.15.28:0/2428803536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 msgr2=0x5646e99580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 0x5646e99580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 -- 172.21.15.28:0/2428803536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e95b0070 msgr2=0x5646e9957650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e95b0070 0x5646e9957650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa217c640 1 -- 172.21.15.28:0/2428803536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e961b650 con 0x5646e95871b0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.165+0000 7fdfa197b640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 0x5646e99580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800a970 tx=0x7fdf98032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486a4b385338b710 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdf90d9a640 1 -- 172.21.15.28:0/2428803536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98032400 con 0x5646e95871b0 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdf937fe640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 0x7fdf880b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdf90d9a640 1 -- 172.21.15.28:0/2428803536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf98005040 con 0x5646e95871b0 2024-09-21T05:49:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdf90d9a640 1 -- 172.21.15.28:0/2428803536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98004a80 con 0x5646e95871b0 2024-09-21T05:49:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdfa297d640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 0x7fdf880b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.166+0000 7fdf937fe640 1 -- 172.21.15.28:0/789392073 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdf8c0e6bb0 con 0x7fdf9c107b30 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 -- 172.21.15.28:0/2428803536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 msgr2=0x5646e9957b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfa297d640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 0x7fdf880b8b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdf94005fd0 tx=0x7fdf94010040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800a970 tx=0x7fdf98032b00 comp rx=0 tx=0).stop 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 -- 172.21.15.28:0/2428803536 shutdown_connections 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e958f390 0x5646e99580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9957b90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2428803536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e95b0070 0x5646e9957650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 -- 172.21.15.28:0/2428803536 >> 172.21.15.28:0/2428803536 conn(0x5646e95a4460 msgr2=0x5646e9955540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 -- 172.21.15.28:0/2428803536 shutdown_connections 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.167+0000 7fdfaa954740 1 -- 172.21.15.28:0/2428803536 wait complete. 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 -- start start 2024-09-21T05:49:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 Processor -- start 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9963250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95b0070 0x5646e996f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5646e971e700 con 0x5646e95871b0 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5646e9858110 con 0x5646e958f390 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5646e9965dd0 con 0x5646e95b0070 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.168+0000 7fdfaa954740 10 client.0 _ll_register_callbacks cb 0x5646e9871990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa297d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95b0070 0x5646e996f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57812/0 (socket says 172.21.15.28:57812) 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 -- 172.21.15.28:0/2643957136 learned_addr learned my addr 172.21.15.28:0/2643957136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa197b640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9963250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 -- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95b0070 msgr2=0x5646e996f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95b0070 0x5646e996f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 -- 172.21.15.28:0/2643957136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 msgr2=0x5646e9963250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9963250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 -- 172.21.15.28:0/2643957136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e9885c40 con 0x5646e95871b0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdfa217c640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005cd0 tx=0x7fdf9802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce26abe26125faec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.169+0000 7fdf85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98052020 con 0x5646e95871b0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdf9804a850 con 0x5646e95871b0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf98005040 con 0x5646e95871b0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdf9805d050 con 0x5646e95871b0 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdfaa954740 10 client.16392 fetch_fsmap learned FSMap version 31 2024-09-21T05:49:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5646e9971a80 con 0x5646e95871b0 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf98005040 con 0x5646e95871b0 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.170+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fdf98044020 con 0x5646e95871b0 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 10 client.16392 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 20 client.16392 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e956e9f0 0x5646e9974de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 4 client.16392 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646e98cfda0 con 0x5646e956e9f0 2024-09-21T05:49:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfa197b640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e956e9f0 0x5646e9974de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdf85ffb640 10 client.16392 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.171+0000 7fdfa197b640 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e956e9f0 0x5646e9974de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f41bee2bdaf72940 server_cookie=4723e8b770787b1d in_seq=0 out_seq=0 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdf85ffb640 1 -- 172.21.15.28:0/2643957136 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5646e98cfda0 con 0x5646e956e9f0 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdf85ffb640 10 client.16392 handle_command_reply: tid=0 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 client.16392 shutdown 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 msgr2=0x5646e9962d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005cd0 tx=0x7fdf9802bd90 comp rx=0 tx=0).stop 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 shutdown_connections 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e956e9f0 0x5646e9974de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95b0070 0x5646e996f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9963250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 --2- 172.21.15.28:0/2643957136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95871b0 0x5646e9962d10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdf85ffb640 0 client.16392 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 >> 172.21.15.28:0/2643957136 conn(0x5646e95a4460 msgr2=0x5646e995a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 shutdown_connections 2024-09-21T05:49:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.172+0000 7fdfaa954740 1 -- 172.21.15.28:0/2643957136 wait complete. 2024-09-21T05:49:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.173+0000 7fdfaa954740 20 client.16392 trim_cache size 0 max 16384 2024-09-21T05:49:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 Processor -- start 2024-09-21T05:49:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 -- start start 2024-09-21T05:49:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e9959040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9964940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9954b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1ad530 con 0x5646e95b0070 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1af160 con 0x5646e958f390 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfaa954740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1adc00 con 0x5646e95871b0 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa297d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9954b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa217c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e9959040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa297d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9954b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57828/0 (socket says 172.21.15.28:57828) 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa297d640 1 -- 172.21.15.28:0/704159954 learned_addr learned my addr 172.21.15.28:0/704159954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa197b640 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9964940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa217c640 1 -- 172.21.15.28:0/704159954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 msgr2=0x5646e9964940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa217c640 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9964940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa217c640 1 -- 172.21.15.28:0/704159954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 msgr2=0x5646e9954b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.191+0000 7fdfa217c640 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9954b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.192+0000 7fdfa217c640 1 -- 172.21.15.28:0/704159954 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e9965450 con 0x5646e95871b0 2024-09-21T05:49:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.192+0000 7fdfa217c640 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e9959040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005d20 tx=0x7fdf98032b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c329ef9cdfbfb16a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.192+0000 7fdf87fff640 1 -- 172.21.15.28:0/704159954 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98052040 con 0x5646e95871b0 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.192+0000 7fdf87fff640 1 -- 172.21.15.28:0/704159954 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf98004b70 con 0x5646e95871b0 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.192+0000 7fdf87fff640 1 -- 172.21.15.28:0/704159954 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9804a850 con 0x5646e95871b0 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- 172.21.15.28:0/704159954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 msgr2=0x5646e9959040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e9959040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf98005d20 tx=0x7fdf98032b00 comp rx=0 tx=0).stop 2024-09-21T05:49:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- 172.21.15.28:0/704159954 shutdown_connections 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9954b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9964940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 --2- 172.21.15.28:0/704159954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e9959040 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- 172.21.15.28:0/704159954 >> 172.21.15.28:0/704159954 conn(0x5646e95a4460 msgr2=0x5646e9964230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- 172.21.15.28:0/704159954 shutdown_connections 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- 172.21.15.28:0/704159954 wait complete. 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.193+0000 7fdfaa954740 1 -- start start 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 Processor -- start 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e996b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9966ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1ae9d0 con 0x5646e95b0070 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1ae2d0 con 0x5646e958f390 2024-09-21T05:49:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c1ab510 con 0x5646e95871b0 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfaa954740 10 client.0 _ll_register_callbacks cb 0x7fdf94009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfa297d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfa297d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57834/0 (socket says 172.21.15.28:57834) 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfa297d640 1 -- 172.21.15.28:0/3548059194 learned_addr learned my addr 172.21.15.28:0/3548059194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfa217c640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e996b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.194+0000 7fdfa197b640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9966ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 -- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 msgr2=0x5646e996b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e996b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 -- 172.21.15.28:0/3548059194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 msgr2=0x5646e9966ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9966ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 -- 172.21.15.28:0/3548059194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e99675e0 con 0x5646e95b0070 2024-09-21T05:49:03.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfa297d640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdf94004750 tx=0x7fdf94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e333a12b1af887 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:03.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdf85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:49:03.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9402c070 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdf94028670 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf98004b70 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdf94038050 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfaa954740 10 client.16398 fetch_fsmap learned FSMap version 31 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.195+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5646e99df3f0 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf98004b70 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8382+0+0 (secure 0 0 0) 0x7fdf94026040 con 0x5646e95b0070 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 10 client.16398 fetch_fsmap finished waiting for FSMap version 31 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 20 client.16398 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e9543da0 0x5646e99e1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 4 client.16398 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:49:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5646e98cb850 con 0x5646e9543da0 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfa217c640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e9543da0 0x5646e99e1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdf85ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.196+0000 7fdfa217c640 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e9543da0 0x5646e99e1cd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdfd54f98dba54ef server_cookie=10286b2fda88de1a in_seq=0 out_seq=0 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdf85ffb640 1 -- 172.21.15.28:0/3548059194 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1996 (crc 0 0 0) 0x5646e98cb850 con 0x5646e9543da0 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdf85ffb640 10 client.16398 handle_command_reply: tid=0 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 client.16398 shutdown 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 msgr2=0x5646e9967020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdf94004750 tx=0x7fdf94004ab0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 shutdown_connections 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5646e9543da0 0x5646e99e1cd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646e95b0070 0x5646e9967020 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5646e958f390 0x5646e9966ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 --2- 172.21.15.28:0/3548059194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5646e95871b0 0x5646e996b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdf85ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 >> 172.21.15.28:0/3548059194 conn(0x5646e95a4460 msgr2=0x5646e9964b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 shutdown_connections 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 1 -- 172.21.15.28:0/3548059194 wait complete. 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.197+0000 7fdfaa954740 20 client.16398 trim_cache size 0 max 16384 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1 inodes in the stack)", 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "45ebe5b7-9e03-4b06-b4b1-cee561706b06", 2024-09-21T05:49:03.203 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.205 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:03.207 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 msgr2=0x7fdf880b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 0x7fdf880b8b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdf94005fd0 tx=0x7fdf94010040 comp rx=0 tx=0).stop 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 msgr2=0x7fdf9c19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c006fd0 tx=0x7fdf8c00b8e0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 shutdown_connections 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdf880b6680 0x7fdf880b8b90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf9c108480 0x7fdf9c19fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c107b30 0x7fdf9c19ab50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 --2- 172.21.15.28:0/789392073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c1021f0 0x7fdf9c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 >> 172.21.15.28:0/789392073 conn(0x7fdf9c077c60 msgr2=0x7fdf9c105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 shutdown_connections 2024-09-21T05:49:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:03.198+0000 7fdfa43e0640 1 -- 172.21.15.28:0/789392073 wait complete. 2024-09-21T05:49:04.447 INFO:tasks.cephfs.filesystem:scrub status for tag:45ebe5b7-9e03-4b06-b4b1-cee561706b06 - {'path': '/', 'tag': '45ebe5b7-9e03-4b06-b4b1-cee561706b06', 'options': 'recursive,force'} 2024-09-21T05:49:04.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:04 smithi181 ceph-mon[34819]: pgmap v386: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 59 MiB/s wr, 8.13k op/s 2024-09-21T05:49:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:04 smithi028 ceph-mon[31227]: pgmap v386: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 59 MiB/s wr, 8.13k op/s 2024-09-21T05:49:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:04 smithi129 ceph-mon[34885]: pgmap v386: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 59 MiB/s wr, 8.13k op/s 2024-09-21T05:49:06.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:06 smithi181 ceph-mon[34819]: pgmap v387: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 43 MiB/s wr, 6.38k op/s 2024-09-21T05:49:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:06 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:49:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:06 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:49:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:06 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:49:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:06 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:06 smithi028 ceph-mon[31227]: pgmap v387: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 43 MiB/s wr, 6.38k op/s 2024-09-21T05:49:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:06 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:49:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:06 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:49:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:06 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:49:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:06 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:06 smithi129 ceph-mon[34885]: pgmap v387: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 43 MiB/s wr, 6.38k op/s 2024-09-21T05:49:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:06 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:49:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:06 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:49:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:06 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:49:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:06 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:07.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:07 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:49:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:07 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:49:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:07 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:49:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:08 smithi028 ceph-mon[31227]: pgmap v388: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 49 MiB/s wr, 7.37k op/s 2024-09-21T05:49:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:08 smithi028 ceph-mon[31227]: Health check update: 3 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:08 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:08 smithi129 ceph-mon[34885]: pgmap v388: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 49 MiB/s wr, 7.37k op/s 2024-09-21T05:49:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:08 smithi129 ceph-mon[34885]: Health check update: 3 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:08 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:08 smithi181 ceph-mon[34819]: pgmap v388: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 49 MiB/s wr, 7.37k op/s 2024-09-21T05:49:08.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:08 smithi181 ceph-mon[34819]: Health check update: 3 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:08.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:08 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:09.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:09 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:09.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:09 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:09.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:09 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:10 smithi028 ceph-mon[31227]: pgmap v389: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 43 MiB/s wr, 6.75k op/s 2024-09-21T05:49:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:10 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:10 smithi129 ceph-mon[34885]: pgmap v389: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 43 MiB/s wr, 6.75k op/s 2024-09-21T05:49:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:10 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:10 smithi181 ceph-mon[34819]: pgmap v389: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 43 MiB/s wr, 6.75k op/s 2024-09-21T05:49:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:10 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:12.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:12 smithi028 ceph-mon[31227]: pgmap v390: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 37 MiB/s wr, 6.03k op/s 2024-09-21T05:49:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:12 smithi129 ceph-mon[34885]: pgmap v390: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 37 MiB/s wr, 6.03k op/s 2024-09-21T05:49:12.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:12 smithi181 ceph-mon[34819]: pgmap v390: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 37 MiB/s wr, 6.03k op/s 2024-09-21T05:49:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:14 smithi129 ceph-mon[34885]: pgmap v391: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 49 MiB/s wr, 7.53k op/s 2024-09-21T05:49:14.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:14 smithi181 ceph-mon[34819]: pgmap v391: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 49 MiB/s wr, 7.53k op/s 2024-09-21T05:49:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:14 smithi028 ceph-mon[31227]: pgmap v391: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 49 MiB/s wr, 7.53k op/s 2024-09-21T05:49:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.013+0000 7f293eff5640 1 -- 172.21.15.28:0/1785655264 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55697c8660f0 con 0x55697c54e1b0 2024-09-21T05:49:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.013+0000 7f293eff5640 10 client.25954 handle_command_reply: tid=0 2024-09-21T05:49:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.013+0000 7f296869c740 1 client.25954 shutdown 2024-09-21T05:49:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 msgr2=0x55697c9050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 0x55697c9050f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2950009230 tx=0x7f2950056690 comp rx=0 tx=0).stop 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 shutdown_connections 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55697c54e1b0 0x55697c97ddb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55697c52d4d0 0x55697c979d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55697c5252f0 0x55697c9050f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 --2- 172.21.15.28:0/1785655264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55697c50cd80 0x55697c904b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 >> 172.21.15.28:0/1785655264 conn(0x55697c542300 msgr2=0x55697c8f1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f293eff5640 0 client.25954 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 shutdown_connections 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 1 -- 172.21.15.28:0/1785655264 wait complete. 2024-09-21T05:49:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.014+0000 7f296869c740 20 client.25954 trim_cache size 0 max 16384 2024-09-21T05:49:15.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:49:15.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:49:15.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:49:15.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 msgr2=0x7f29440b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 0x7f29440b8bc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f295002aa00 tx=0x7f2950005780 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 msgr2=0x7f295c103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f295400efd0 tx=0x7f295400c6a0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 shutdown_connections 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f29440b66b0 0x7f29440b8bc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f295c109d80 0x7f295c103a70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295c109430 0x7f295c103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 --2- 172.21.15.28:0/3415817642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f295c102060 0x7f295c102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 >> 172.21.15.28:0/3415817642 conn(0x7f295c079e90 msgr2=0x7f295c1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 shutdown_connections 2024-09-21T05:49:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:15.015+0000 7f296172b640 1 -- 172.21.15.28:0/3415817642 wait complete. 2024-09-21T05:49:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:16 smithi129 ceph-mon[34885]: pgmap v392: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 37 MiB/s wr, 5.04k op/s 2024-09-21T05:49:16.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:16 smithi181 ceph-mon[34819]: pgmap v392: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 37 MiB/s wr, 5.04k op/s 2024-09-21T05:49:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:16 smithi028 ceph-mon[31227]: pgmap v392: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 37 MiB/s wr, 5.04k op/s 2024-09-21T05:49:17.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:17 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:17.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:17 smithi129 ceph-mon[34885]: Health check update: 4 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:17.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:17 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:17.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:17 smithi181 ceph-mon[34819]: Health check update: 4 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:17 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:17.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:17 smithi028 ceph-mon[31227]: Health check update: 4 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:18 smithi129 ceph-mon[34885]: pgmap v393: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:49:18.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:18 smithi181 ceph-mon[34819]: pgmap v393: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:49:18.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:18 smithi028 ceph-mon[31227]: pgmap v393: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.73k op/s 2024-09-21T05:49:20.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:20 smithi181 ceph-mon[34819]: pgmap v394: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 46 MiB/s wr, 5.20k op/s 2024-09-21T05:49:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:20 smithi028 ceph-mon[31227]: pgmap v394: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 46 MiB/s wr, 5.20k op/s 2024-09-21T05:49:20.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:20 smithi129 ceph-mon[34885]: pgmap v394: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 46 MiB/s wr, 5.20k op/s 2024-09-21T05:49:21.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:21 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:21.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:21 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:21.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:21 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:49:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:22 smithi181 ceph-mon[34819]: pgmap v395: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 43 MiB/s wr, 4.70k op/s 2024-09-21T05:49:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:22 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:22 smithi028 ceph-mon[31227]: pgmap v395: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 43 MiB/s wr, 4.70k op/s 2024-09-21T05:49:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:22 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:22.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:22 smithi129 ceph-mon[34885]: pgmap v395: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 43 MiB/s wr, 4.70k op/s 2024-09-21T05:49:22.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:22 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:24.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:24 smithi181 ceph-mon[34819]: pgmap v396: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 72 MiB/s wr, 7.86k op/s 2024-09-21T05:49:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:24 smithi028 ceph-mon[31227]: pgmap v396: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 72 MiB/s wr, 7.86k op/s 2024-09-21T05:49:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:24 smithi129 ceph-mon[34885]: pgmap v396: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 72 MiB/s wr, 7.86k op/s 2024-09-21T05:49:26.666 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:26 smithi129 ceph-mon[34885]: pgmap v397: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 69 MiB/s wr, 7.47k op/s 2024-09-21T05:49:26.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:26 smithi181 ceph-mon[34819]: pgmap v397: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 69 MiB/s wr, 7.47k op/s 2024-09-21T05:49:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:26 smithi028 ceph-mon[31227]: pgmap v397: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 69 MiB/s wr, 7.47k op/s 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 -- 172.21.15.28:0/2733968541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501047c0 msgr2=0x7f7150104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 --2- 172.21.15.28:0/2733968541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501047c0 0x7f7150104bd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7140009900 tx=0x7f714002a410 comp rx=0 tx=0).stop 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 -- 172.21.15.28:0/2733968541 shutdown_connections 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 --2- 172.21.15.28:0/2733968541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f715010dd70 0x7f7150110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 --2- 172.21.15.28:0/2733968541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71501051f0 0x7f715010d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 --2- 172.21.15.28:0/2733968541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501047c0 0x7f7150104bd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 -- 172.21.15.28:0/2733968541 >> 172.21.15.28:0/2733968541 conn(0x7f715007a6f0 msgr2=0x7f715007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 -- 172.21.15.28:0/2733968541 shutdown_connections 2024-09-21T05:49:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.144+0000 7f7154f19640 1 -- 172.21.15.28:0/2733968541 wait complete. 2024-09-21T05:49:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.145+0000 7f7154f19640 1 Processor -- start 2024-09-21T05:49:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.145+0000 7f7154f19640 1 -- start start 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501051f0 0x7f715019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f715010dd70 0x7f715019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f714e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f714e59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46988/0 (socket says 172.21.15.28:46988) 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f714dd9b640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501051f0 0x7f715019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f714e59c640 1 -- 172.21.15.28:0/1631567844 learned_addr learned my addr 172.21.15.28:0/1631567844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f714ed9d640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f715010dd70 0x7f715019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7150113a10 con 0x7f71501051f0 2024-09-21T05:49:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7150113890 con 0x7f715010dd70 2024-09-21T05:49:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.146+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7150113b90 con 0x7f71501047c0 2024-09-21T05:49:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 -- 172.21.15.28:0/1631567844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f715010dd70 msgr2=0x7f715019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f715010dd70 0x7f715019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 -- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501051f0 msgr2=0x7f715019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501051f0 0x7f715019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71501a03c0 con 0x7f71501047c0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f714e59c640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7140005940 tx=0x7f71400040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f713f7fe640 1 -- 172.21.15.28:0/1631567844 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714003f030 con 0x7f71501047c0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f713f7fe640 1 -- 172.21.15.28:0/1631567844 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71400044c0 con 0x7f71501047c0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.147+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71501a0650 con 0x7f71501047c0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.148+0000 7f713f7fe640 1 -- 172.21.15.28:0/1631567844 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140035070 con 0x7f71501047c0 2024-09-21T05:49:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.148+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71501a6e40 con 0x7f71501047c0 2024-09-21T05:49:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.151+0000 7f715be83740 1 Processor -- start 2024-09-21T05:49:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.151+0000 7f715be83740 1 -- start start 2024-09-21T05:49:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c351f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c351f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c3516c6f0 con 0x559c34e314d0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559c3515bb20 con 0x559c34e521b0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f715be83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559c3515f930 con 0x559c34e292f0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f713f7fe640 1 -- 172.21.15.28:0/1631567844 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7140042030 con 0x7f71501047c0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37508/0 (socket says 172.21.15.28:37508) 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c351f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 -- 172.21.15.28:0/4115352280 learned_addr learned my addr 172.21.15.28:0/4115352280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714e59c640 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c351f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 -- 172.21.15.28:0/4115352280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 msgr2=0x559c351f7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c351f7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 -- 172.21.15.28:0/4115352280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 msgr2=0x559c351f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c351f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.152+0000 7f714ed9d640 1 -- 172.21.15.28:0/4115352280 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c35121530 con 0x559c34e521b0 2024-09-21T05:49:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f714ed9d640 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7148009f50 tx=0x7f71480118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb4a73427cbb6fe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f713cff9640 1 -- 172.21.15.28:0/4115352280 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148004d20 con 0x559c34e521b0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f713cff9640 1 -- 172.21.15.28:0/4115352280 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7148004050 con 0x559c34e521b0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f713f7fe640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 0x7f71380b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f714e59c640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 0x7f71380b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f713cff9640 1 -- 172.21.15.28:0/4115352280 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148005280 con 0x559c34e521b0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.153+0000 7f713f7fe640 1 -- 172.21.15.28:0/1631567844 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7140107a00 con 0x7f71501047c0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 -- 172.21.15.28:0/4115352280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 msgr2=0x559c351f79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f714e59c640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 0x7f71380b8c10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7140105220 tx=0x7f714003b4f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7148009f50 tx=0x7f71480118d0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 -- 172.21.15.28:0/4115352280 shutdown_connections 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c351f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c351f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 --2- 172.21.15.28:0/4115352280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c351f79b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 -- 172.21.15.28:0/4115352280 >> 172.21.15.28:0/4115352280 conn(0x559c34e46300 msgr2=0x559c351f58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 -- 172.21.15.28:0/4115352280 shutdown_connections 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.154+0000 7f715be83740 1 -- 172.21.15.28:0/4115352280 wait complete. 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.155+0000 7f715be83740 1 -- start start 2024-09-21T05:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.155+0000 7f715be83740 1 Processor -- start 2024-09-21T05:49:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.155+0000 7f715be83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c352073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.155+0000 7f715be83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c352078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c35207e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c35093ab0 con 0x559c34e314d0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559c35101730 con 0x559c34e521b0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559c3520a560 con 0x559c34e292f0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c352078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c352073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c352073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46996/0 (socket says 172.21.15.28:46996) 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714ed9d640 1 -- 172.21.15.28:0/351770828 learned_addr learned my addr 172.21.15.28:0/351770828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714ed9d640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c35207e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 -- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 msgr2=0x559c352073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c352073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 -- 172.21.15.28:0/351770828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 msgr2=0x559c35207e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c35207e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 -- 172.21.15.28:0/351770828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c34ebd790 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 10 client.0 _ll_register_callbacks cb 0x559c351556b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7148004050 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.156+0000 7f714dd9b640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c352078f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f715006c6f0 tx=0x7f7144012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ad0b4d46cfacd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f7135ffb640 10 client.16410 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f7135ffb640 1 -- 172.21.15.28:0/351770828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714400f8f0 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f7135ffb640 1 -- 172.21.15.28:0/351770828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f714400fa90 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f7135ffb640 1 -- 172.21.15.28:0/351770828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f714400fcf0 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f715be83740 10 client.16410 fetch_fsmap learned FSMap version 33 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=33}) -- 0x559c34f36620 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.157+0000 7f7135ffb640 1 -- 172.21.15.28:0/351770828 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f7144026470 con 0x559c34e314d0 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 10 client.16410 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 20 client.16410 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34e10d80 0x559c352194b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 4 client.16410 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c3516e640 con 0x559c34e10d80 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f714ed9d640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34e10d80 0x559c352194b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f714ed9d640 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34e10d80 0x559c352194b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22278c951c5f8867 server_cookie=ed9e231a9bb07b08 in_seq=0 out_seq=0 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.158+0000 7f7135ffb640 10 client.16410 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f7135ffb640 1 -- 172.21.15.28:0/351770828 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559c3516e640 con 0x559c34e10d80 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f7135ffb640 10 client.16410 handle_command_reply: tid=0 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 client.16410 shutdown 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 msgr2=0x559c352078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c352078f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f715006c6f0 tx=0x7f7144012b60 comp rx=0 tx=0).stop 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 shutdown_connections 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34e10d80 0x559c352194b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e521b0 0x559c35207e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c352078f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 --2- 172.21.15.28:0/351770828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e292f0 0x559c352073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 >> 172.21.15.28:0/351770828 conn(0x559c34e46300 msgr2=0x559c351f93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f7135ffb640 0 client.16410 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.159+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 shutdown_connections 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.160+0000 7f715be83740 1 -- 172.21.15.28:0/351770828 wait complete. 2024-09-21T05:49:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.160+0000 7f715be83740 20 client.16410 trim_cache size 0 max 16384 2024-09-21T05:49:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.177+0000 7f715be83740 1 Processor -- start 2024-09-21T05:49:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.177+0000 7f715be83740 1 -- start start 2024-09-21T05:49:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.177+0000 7f715be83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c35204580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f715be83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f715be83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e521b0 0x559c3521ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f715be83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71501ad460 con 0x559c34e314d0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f715be83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71501af0f0 con 0x559c34e292f0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f715be83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71501adb30 con 0x559c34e521b0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41400/0 (socket says 172.21.15.28:41400) 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 -- 172.21.15.28:0/2735250773 learned_addr learned my addr 172.21.15.28:0/2735250773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e521b0 0x559c3521ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c35204580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 -- 172.21.15.28:0/2735250773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e521b0 msgr2=0x559c3521ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e521b0 0x559c3521ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 -- 172.21.15.28:0/2735250773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 msgr2=0x559c35204580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c35204580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 -- 172.21.15.28:0/2735250773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c35219440 con 0x559c34e314d0 2024-09-21T05:49:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f714dd9b640 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f71440057d0 tx=0x7f7144014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03f107e17adeffc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f7137fff640 1 -- 172.21.15.28:0/2735250773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144026470 con 0x559c34e314d0 2024-09-21T05:49:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f7137fff640 1 -- 172.21.15.28:0/2735250773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f714402c020 con 0x559c34e314d0 2024-09-21T05:49:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.178+0000 7f7137fff640 1 -- 172.21.15.28:0/2735250773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144026bd0 con 0x559c34e314d0 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 -- 172.21.15.28:0/2735250773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 msgr2=0x559c35204ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f71440057d0 tx=0x7f7144014000 comp rx=0 tx=0).stop 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 -- 172.21.15.28:0/2735250773 shutdown_connections 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e521b0 0x559c3521ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e314d0 0x559c35204ac0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 --2- 172.21.15.28:0/2735250773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c35204580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 -- 172.21.15.28:0/2735250773 >> 172.21.15.28:0/2735250773 conn(0x559c34e46300 msgr2=0x559c351fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 -- 172.21.15.28:0/2735250773 shutdown_connections 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.179+0000 7f715be83740 1 -- 172.21.15.28:0/2735250773 wait complete. 2024-09-21T05:49:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 -- start start 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 Processor -- start 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c3520cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e314d0 0x559c3520d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71501ae960 con 0x559c34e521b0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71501ae260 con 0x559c34e292f0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71501ab440 con 0x559c34e314d0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f715be83740 10 client.0 _ll_register_callbacks cb 0x7f7144009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f714ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41408/0 (socket says 172.21.15.28:41408) 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f714dd9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e314d0 0x559c3520d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f714ed9d640 1 -- 172.21.15.28:0/2326254713 learned_addr learned my addr 172.21.15.28:0/2326254713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.180+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c3520cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 -- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e314d0 msgr2=0x559c3520d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e314d0 0x559c3520d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 -- 172.21.15.28:0/2326254713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 msgr2=0x559c3520cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c3520cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 -- 172.21.15.28:0/2326254713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c35282420 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f714ed9d640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7148003d80 tx=0x7f714802ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e0f6a2d93f7ff9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148028450 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f71480285f0 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f714402c020 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f71480287f0 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f715be83740 10 client.16416 fetch_fsmap learned FSMap version 33 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559c35283670 con 0x559c34e521b0 2024-09-21T05:49:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f714402c020 con 0x559c34e521b0 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.181+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8390+0+0 (secure 0 0 0) 0x7f7148043050 con 0x559c34e521b0 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 10 client.16416 fetch_fsmap finished waiting for FSMap version 33 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 20 client.16416 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34de61c0 0x559c35285ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 4 client.16416 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c3516a0f0 con 0x559c34de61c0 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f714dd9b640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34de61c0 0x559c35285ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f714dd9b640 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34de61c0 0x559c35285ca0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533a0f6744abc663 server_cookie=3a10d96d6822a103 in_seq=0 out_seq=0 2024-09-21T05:49:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:49:28.182+0000 7f7135ffb640 10 client.16416 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:28.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:28 smithi181 ceph-mon[34819]: pgmap v398: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 MiB/s wr, 9.19k op/s 2024-09-21T05:49:28.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:28 smithi028 ceph-mon[31227]: pgmap v398: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 MiB/s wr, 9.19k op/s 2024-09-21T05:49:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:28 smithi129 ceph-mon[34885]: pgmap v398: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 MiB/s wr, 9.19k op/s 2024-09-21T05:49:29.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:29 smithi181 ceph-mon[34819]: Health check update: 6 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:29.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:29 smithi028 ceph-mon[31227]: Health check update: 6 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:29 smithi129 ceph-mon[34885]: Health check update: 6 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:30.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:30 smithi181 ceph-mon[34819]: pgmap v399: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 MiB/s wr, 9.12k op/s 2024-09-21T05:49:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:30 smithi028 ceph-mon[31227]: pgmap v399: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 MiB/s wr, 9.12k op/s 2024-09-21T05:49:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:30 smithi129 ceph-mon[34885]: pgmap v399: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 MiB/s wr, 9.12k op/s 2024-09-21T05:49:32.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:32 smithi181 ceph-mon[34819]: pgmap v400: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 73 MiB/s wr, 8.36k op/s 2024-09-21T05:49:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:32 smithi028 ceph-mon[31227]: pgmap v400: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 73 MiB/s wr, 8.36k op/s 2024-09-21T05:49:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:32 smithi129 ceph-mon[34885]: pgmap v400: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 73 MiB/s wr, 8.36k op/s 2024-09-21T05:49:34.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:49:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:49:34.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:34 smithi181 ceph-mon[34819]: pgmap v401: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 87 MiB/s wr, 10.19k op/s 2024-09-21T05:49:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:34 smithi028 ceph-mon[31227]: pgmap v401: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 87 MiB/s wr, 10.19k op/s 2024-09-21T05:49:34.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:34 smithi129 ceph-mon[34885]: pgmap v401: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 87 MiB/s wr, 10.19k op/s 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 -- 172.21.15.28:0/2443389091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79681083a0 msgr2=0x7f79681087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 --2- 172.21.15.28:0/2443389091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79681083a0 0x7f79681087b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802a410 comp rx=0 tx=0).stop 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 -- 172.21.15.28:0/2443389091 shutdown_connections 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 --2- 172.21.15.28:0/2443389091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968108cf0 0x7f796810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 --2- 172.21.15.28:0/2443389091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79681083a0 0x7f79681087b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 --2- 172.21.15.28:0/2443389091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7968102a60 0x7f7968102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 -- 172.21.15.28:0/2443389091 >> 172.21.15.28:0/2443389091 conn(0x7f7968079730 msgr2=0x7f7968079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 -- 172.21.15.28:0/2443389091 shutdown_connections 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.285+0000 7f797021b640 1 -- 172.21.15.28:0/2443389091 wait complete. 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 Processor -- start 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 -- start start 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79681083a0 0x7f796819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79681129b0 con 0x7f7968108cf0 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7968112830 con 0x7f79681083a0 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f797021b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7968112b30 con 0x7f7968102a60 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796e7b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57558/0 (socket says 172.21.15.28:57558) 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796e7b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47022/0 (socket says 172.21.15.28:47022) 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796dfb7640 1 -- 172.21.15.28:0/1807480547 learned_addr learned my addr 172.21.15.28:0/1807480547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.286+0000 7f796d7b6640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79681083a0 0x7f796819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 -- 172.21.15.28:0/1807480547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79681083a0 msgr2=0x7f796819b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79681083a0 0x7f796819b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 -- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 msgr2=0x7f796819bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 -- 172.21.15.28:0/1807480547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79681a65c0 con 0x7f7968102a60 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796e7b8640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f796400ea60 tx=0x7f796400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f795effd640 1 -- 172.21.15.28:0/1807480547 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796400cde0 con 0x7f7968102a60 2024-09-21T05:49:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681a68b0 con 0x7f7968102a60 2024-09-21T05:49:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f795effd640 1 -- 172.21.15.28:0/1807480547 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79640035f0 con 0x7f7968102a60 2024-09-21T05:49:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f795effd640 1 -- 172.21.15.28:0/1807480547 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964010690 con 0x7f7968102a60 2024-09-21T05:49:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.287+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681a6e10 con 0x7f7968102a60 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.290+0000 7f795effd640 1 -- 172.21.15.28:0/1807480547 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f79640037a0 con 0x7f7968102a60 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.290+0000 7f797678f740 1 Processor -- start 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 -- start start 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c01b0 0x5583bd890b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c8390 0x5583bd8910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583bd806e50 con 0x5583bd4c8390 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5583bd7f6280 con 0x5583bd4e9070 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f797678f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5583bd7fa090 con 0x5583bd4c01b0 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f796e7b8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c01b0 0x5583bd890b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f796e7b8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55864/0 (socket says 172.21.15.28:55864) 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f796e7b8640 1 -- 172.21.15.28:0/2665900808 learned_addr learned my addr 172.21.15.28:0/2665900808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.291+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c8390 0x5583bd8910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f795effd640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 0x7f79540b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 -- 172.21.15.28:0/2665900808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c01b0 msgr2=0x5583bd890b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c01b0 0x5583bd890b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 -- 172.21.15.28:0/2665900808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c8390 msgr2=0x5583bd8910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c8390 0x5583bd8910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 -- 172.21.15.28:0/2665900808 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583bd554650 con 0x5583bd4e9070 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f795effd640 1 -- 172.21.15.28:0/1807480547 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f79640e79e0 con 0x7f7968102a60 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7960011790 tx=0x7f7960011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a954ee6968960b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.292+0000 7f796e7b8640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 0x7f79540b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.293+0000 7f7953fff640 1 -- 172.21.15.28:0/2665900808 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960003b30 con 0x5583bd4e9070 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.293+0000 7f7953fff640 1 -- 172.21.15.28:0/2665900808 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796000fe50 con 0x5583bd4e9070 2024-09-21T05:49:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.293+0000 7f7953fff640 1 -- 172.21.15.28:0/2665900808 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796001a750 con 0x5583bd4e9070 2024-09-21T05:49:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- 172.21.15.28:0/2665900808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 msgr2=0x5583bd890650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f796e7b8640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 0x7f79540b8b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f796001c040 tx=0x7f796001aaf0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7960011790 tx=0x7f7960011c50 comp rx=0 tx=0).stop 2024-09-21T05:49:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- 172.21.15.28:0/2665900808 shutdown_connections 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c8390 0x5583bd8910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c01b0 0x5583bd890b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 --2- 172.21.15.28:0/2665900808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd890650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- 172.21.15.28:0/2665900808 >> 172.21.15.28:0/2665900808 conn(0x5583bd4dd460 msgr2=0x5583bd88e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- 172.21.15.28:0/2665900808 shutdown_connections 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- 172.21.15.28:0/2665900808 wait complete. 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 -- start start 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.294+0000 7f797678f740 1 Processor -- start 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd8a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583bd657700 con 0x5583bd4c01b0 2024-09-21T05:49:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5583bd791110 con 0x5583bd4c8390 2024-09-21T05:49:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5583bd89ee30 con 0x5583bd4e9070 2024-09-21T05:49:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f797678f740 10 client.0 _ll_register_callbacks cb 0x5583bd7aa990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47042/0 (socket says 172.21.15.28:47042) 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796dfb7640 1 -- 172.21.15.28:0/4083602407 learned_addr learned my addr 172.21.15.28:0/4083602407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796e7b8640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd8a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796d7b6640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796dfb7640 1 -- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 msgr2=0x5583bd8a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.295+0000 7f796dfb7640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd8a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f796dfb7640 1 -- 172.21.15.28:0/4083602407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 msgr2=0x5583bd89c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f796dfb7640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f796dfb7640 1 -- 172.21.15.28:0/4083602407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583bd7bec40 con 0x5583bd4c01b0 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f796dfb7640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7964009920 tx=0x7f79640eb120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de72c6b708e39e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f79517fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79640a9ea0 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7964004200 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f796000fe50 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f79640acfc0 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f797678f740 10 client.16428 fetch_fsmap learned FSMap version 34 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5583bd8aab60 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.296+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796000fe50 con 0x5583bd4c01b0 2024-09-21T05:49:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f79640aeb80 con 0x5583bd4c01b0 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 10 client.16428 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 20 client.16428 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd4a79f0 0x5583bd8ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 4 client.16428 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:49:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583bd808da0 con 0x5583bd4a79f0 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.297+0000 7f796d7b6640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd4a79f0 0x5583bd8ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.298+0000 7f79517fa640 10 client.16428 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.298+0000 7f796d7b6640 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd4a79f0 0x5583bd8ae0d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a213071a97c68b server_cookie=3628721a5a154603 in_seq=0 out_seq=0 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f79517fa640 1 -- 172.21.15.28:0/4083602407 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5583bd808da0 con 0x5583bd4a79f0 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f79517fa640 10 client.16428 handle_command_reply: tid=0 2024-09-21T05:49:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 client.16428 shutdown 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 msgr2=0x5583bd89bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7964009920 tx=0x7f79640eb120 comp rx=0 tx=0).stop 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 shutdown_connections 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd4a79f0 0x5583bd8ae0d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd8a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 --2- 172.21.15.28:0/4083602407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd89bcd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f79517fa640 0 client.16428 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 >> 172.21.15.28:0/4083602407 conn(0x5583bd4dd460 msgr2=0x5583bd893830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 shutdown_connections 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 1 -- 172.21.15.28:0/4083602407 wait complete. 2024-09-21T05:49:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.299+0000 7f797678f740 20 client.16428 trim_cache size 0 max 16384 2024-09-21T05:49:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.318+0000 7f797678f740 1 Processor -- start 2024-09-21T05:49:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.318+0000 7f797678f740 1 -- start start 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd898ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79681af820 con 0x5583bd4c01b0 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b14b0 con 0x5583bd4c8390 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f797678f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79681afef0 con 0x5583bd4e9070 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47048/0 (socket says 172.21.15.28:47048) 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796dfb7640 1 -- 172.21.15.28:0/2412844718 learned_addr learned my addr 172.21.15.28:0/2412844718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd898ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 -- 172.21.15.28:0/2412844718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 msgr2=0x5583bd898ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd898ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 -- 172.21.15.28:0/2412844718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 msgr2=0x5583bd892040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796d7b6640 1 -- 172.21.15.28:0/2412844718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583bd89cf90 con 0x5583bd4c8390 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796dfb7640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.319+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd898ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.320+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89d900 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f79580045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8586e997228a1f28 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.320+0000 7f79537fe640 1 -- 172.21.15.28:0/2412844718 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795804a020 con 0x5583bd4c8390 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.320+0000 7f79537fe640 1 -- 172.21.15.28:0/2412844718 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7958004750 con 0x5583bd4c8390 2024-09-21T05:49:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.320+0000 7f79537fe640 1 -- 172.21.15.28:0/2412844718 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958035040 con 0x5583bd4c8390 2024-09-21T05:49:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- 172.21.15.28:0/2412844718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 msgr2=0x5583bd89d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89d900 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f79580045e0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- 172.21.15.28:0/2412844718 shutdown_connections 2024-09-21T05:49:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4e9070 0x5583bd898ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4c8390 0x5583bd89d900 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 --2- 172.21.15.28:0/2412844718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd892040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- 172.21.15.28:0/2412844718 >> 172.21.15.28:0/2412844718 conn(0x5583bd4dd460 msgr2=0x5583bd89d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- 172.21.15.28:0/2412844718 shutdown_connections 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- 172.21.15.28:0/2412844718 wait complete. 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 -- start start 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.321+0000 7f797678f740 1 Processor -- start 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd8a4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd8a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b0d20 con 0x5583bd4c01b0 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b0620 con 0x5583bd4e9070 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79681ad800 con 0x5583bd4c8390 2024-09-21T05:49:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f797678f740 10 client.0 _ll_register_callbacks cb 0x7f7964003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f796d7b6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f796d7b6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57604/0 (socket says 172.21.15.28:57604) 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f796d7b6640 1 -- 172.21.15.28:0/2232875006 learned_addr learned my addr 172.21.15.28:0/2232875006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f796dfb7640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd8a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.322+0000 7f796e7b8640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd8a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 -- 172.21.15.28:0/2232875006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 msgr2=0x5583bd8a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd8a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 -- 172.21.15.28:0/2232875006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 msgr2=0x5583bd8a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd8a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 -- 172.21.15.28:0/2232875006 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583bd8a2a20 con 0x5583bd4c8390 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f796d7b6640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7958009850 tx=0x7f795802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=312fbb5a24abebc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f79517fa640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795804a050 con 0x5583bd4c8390 2024-09-21T05:49:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7958004750 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7958003c40 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f795805e050 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f797678f740 10 client.26035 fetch_fsmap learned FSMap version 34 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.323+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5583bd91cde0 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7958004750 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f795803d570 con 0x5583bd4c8390 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 10 client.26035 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 10 client.26035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 20 client.26035 populate_metadata read hostname 'smithi028' 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd47cda0 0x5583bd91f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 4 client.26035 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5583bd804850 con 0x5583bd47cda0 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f796dfb7640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd47cda0 0x5583bd91f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f79517fa640 10 client.26035 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.324+0000 7f796dfb7640 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd47cda0 0x5583bd91f600 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7c960112c839e9 server_cookie=1f87bfd89469e1b9 in_seq=0 out_seq=0 2024-09-21T05:49:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f79517fa640 1 -- 172.21.15.28:0/2232875006 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1990 (crc 0 0 0) 0x5583bd804850 con 0x5583bd47cda0 2024-09-21T05:49:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f79517fa640 10 client.26035 handle_command_reply: tid=0 2024-09-21T05:49:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 client.26035 shutdown 2024-09-21T05:49:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 msgr2=0x5583bd8a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7958009850 tx=0x7f795802a410 comp rx=0 tx=0).stop 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 shutdown_connections 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5583bd47cda0 0x5583bd91f600 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583bd4e9070 0x5583bd8a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583bd4c8390 0x5583bd8a1d70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f79517fa640 0 client.26035 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 --2- 172.21.15.28:0/2232875006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583bd4c01b0 0x5583bd8a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 >> 172.21.15.28:0/2232875006 conn(0x5583bd4dd460 msgr2=0x5583bd893ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 shutdown_connections 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 1 -- 172.21.15.28:0/2232875006 wait complete. 2024-09-21T05:49:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.325+0000 7f797678f740 20 client.26035 trim_cache size 0 max 16384 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "45ebe5b7-9e03-4b06-b4b1-cee561706b06", 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:49:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:49:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 msgr2=0x7f79540b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 0x7f79540b8b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f796001c040 tx=0x7f796001aaf0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 msgr2=0x7f796819b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f796400ea60 tx=0x7f796400ef20 comp rx=0 tx=0).stop 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 shutdown_connections 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f79540b6680 0x7f79540b8b90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968108cf0 0x7f796819bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79681083a0 0x7f796819b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 --2- 172.21.15.28:0/1807480547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7968102a60 0x7f796819b300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 >> 172.21.15.28:0/1807480547 conn(0x7f7968079730 msgr2=0x7f7968104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 shutdown_connections 2024-09-21T05:49:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:49:35.326+0000 7f797021b640 1 -- 172.21.15.28:0/1807480547 wait complete. 2024-09-21T05:49:35.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:35 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:35.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:35 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:35.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:35 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:49:36.163 INFO:tasks.cephfs.filesystem:scrub status for tag:45ebe5b7-9e03-4b06-b4b1-cee561706b06 - {'path': '/', 'tag': '45ebe5b7-9e03-4b06-b4b1-cee561706b06', 'options': 'recursive,force'} 2024-09-21T05:49:36.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:36 smithi181 ceph-mon[34819]: pgmap v402: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 62 MiB/s wr, 7.55k op/s 2024-09-21T05:49:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:36 smithi028 ceph-mon[31227]: pgmap v402: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 62 MiB/s wr, 7.55k op/s 2024-09-21T05:49:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:36 smithi129 ceph-mon[34885]: pgmap v402: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 62 MiB/s wr, 7.55k op/s 2024-09-21T05:49:37.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:37 smithi181 ceph-mon[34819]: Health check update: 7 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:37.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:37 smithi028 ceph-mon[31227]: Health check update: 7 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:37.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:37 smithi129 ceph-mon[34885]: Health check update: 7 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T05:49:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:38 smithi181 ceph-mon[34819]: pgmap v403: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 MiB/s wr, 7.49k op/s 2024-09-21T05:49:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:38 smithi028 ceph-mon[31227]: pgmap v403: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 MiB/s wr, 7.49k op/s 2024-09-21T05:49:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:38 smithi129 ceph-mon[34885]: pgmap v403: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 MiB/s wr, 7.49k op/s 2024-09-21T05:49:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:40 smithi181 ceph-mon[34819]: pgmap v404: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 48 MiB/s wr, 6.22k op/s 2024-09-21T05:49:40.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:40 smithi028 ceph-mon[31227]: pgmap v404: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 48 MiB/s wr, 6.22k op/s 2024-09-21T05:49:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:40 smithi129 ceph-mon[34885]: pgmap v404: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 48 MiB/s wr, 6.22k op/s 2024-09-21T05:49:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:42 smithi181 ceph-mon[34819]: pgmap v405: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 41 MiB/s wr, 5.35k op/s 2024-09-21T05:49:42.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:42 smithi028 ceph-mon[31227]: pgmap v405: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 41 MiB/s wr, 5.35k op/s 2024-09-21T05:49:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:42 smithi129 ceph-mon[34885]: pgmap v405: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 41 MiB/s wr, 5.35k op/s 2024-09-21T05:49:44.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:44 smithi181 ceph-mon[34819]: pgmap v406: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 57 MiB/s wr, 7.24k op/s 2024-09-21T05:49:44.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:44 smithi028 ceph-mon[31227]: pgmap v406: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 57 MiB/s wr, 7.24k op/s 2024-09-21T05:49:44.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:44 smithi129 ceph-mon[34885]: pgmap v406: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 57 MiB/s wr, 7.24k op/s 2024-09-21T05:49:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:46 smithi028 ceph-mon[31227]: pgmap v407: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 MiB/s wr, 5.96k op/s 2024-09-21T05:49:46.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:46 smithi181 ceph-mon[34819]: pgmap v407: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 MiB/s wr, 5.96k op/s 2024-09-21T05:49:46.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:46 smithi129 ceph-mon[34885]: pgmap v407: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 MiB/s wr, 5.96k op/s 2024-09-21T05:49:48.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:48 smithi181 ceph-mon[34819]: pgmap v408: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 6.50k op/s 2024-09-21T05:49:48.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:48 smithi028 ceph-mon[31227]: pgmap v408: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 6.50k op/s 2024-09-21T05:49:48.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:48 smithi129 ceph-mon[34885]: pgmap v408: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 6.50k op/s 2024-09-21T05:49:49.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:49:49.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:49:49.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:49:50.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:50 smithi181 ceph-mon[34819]: pgmap v409: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 6.10k op/s 2024-09-21T05:49:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:49:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:49:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:49:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:50 smithi028 ceph-mon[31227]: pgmap v409: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 6.10k op/s 2024-09-21T05:49:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:49:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:49:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:49:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:50 smithi129 ceph-mon[34885]: pgmap v409: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 6.10k op/s 2024-09-21T05:49:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:49:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:49:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:49:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:52 smithi181 ceph-mon[34819]: pgmap v410: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 MiB/s wr, 5.40k op/s 2024-09-21T05:49:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:52 smithi028 ceph-mon[31227]: pgmap v410: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 MiB/s wr, 5.40k op/s 2024-09-21T05:49:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:52 smithi129 ceph-mon[34885]: pgmap v410: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 MiB/s wr, 5.40k op/s 2024-09-21T05:49:54.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:54 smithi181 ceph-mon[34819]: pgmap v411: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 58 MiB/s wr, 7.04k op/s 2024-09-21T05:49:54.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:54 smithi028 ceph-mon[31227]: pgmap v411: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 58 MiB/s wr, 7.04k op/s 2024-09-21T05:49:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:54 smithi129 ceph-mon[34885]: pgmap v411: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 58 MiB/s wr, 7.04k op/s 2024-09-21T05:49:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:56 smithi181 ceph-mon[34819]: pgmap v412: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 5.43k op/s 2024-09-21T05:49:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:56 smithi028 ceph-mon[31227]: pgmap v412: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 5.43k op/s 2024-09-21T05:49:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:56 smithi129 ceph-mon[34885]: pgmap v412: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 5.43k op/s 2024-09-21T05:49:58.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:49:58 smithi181 ceph-mon[34819]: pgmap v413: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 51 MiB/s wr, 6.30k op/s 2024-09-21T05:49:58.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:49:58 smithi028 ceph-mon[31227]: pgmap v413: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 51 MiB/s wr, 6.30k op/s 2024-09-21T05:49:58.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:49:58 smithi129 ceph-mon[34885]: pgmap v413: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 51 MiB/s wr, 6.30k op/s 2024-09-21T05:50:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: pgmap v414: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 49 MiB/s wr, 6.09k op/s 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: Health detail: HEALTH_WARN 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: [WRN] BLUESTORE_SLOW_OP_ALERT: 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.1 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.2 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.4 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.5 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.8 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.10 observed slow operation indications in BlueStore 2024-09-21T05:50:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:00 smithi181 ceph-mon[34819]: osd.11 observed slow operation indications in BlueStore 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: pgmap v414: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 49 MiB/s wr, 6.09k op/s 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: Health detail: HEALTH_WARN 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: [WRN] BLUESTORE_SLOW_OP_ALERT: 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.1 observed slow operation indications in BlueStore 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.2 observed slow operation indications in BlueStore 2024-09-21T05:50:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.4 observed slow operation indications in BlueStore 2024-09-21T05:50:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.5 observed slow operation indications in BlueStore 2024-09-21T05:50:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.8 observed slow operation indications in BlueStore 2024-09-21T05:50:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.10 observed slow operation indications in BlueStore 2024-09-21T05:50:00.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:00 smithi028 ceph-mon[31227]: osd.11 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: pgmap v414: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 49 MiB/s wr, 6.09k op/s 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: Health detail: HEALTH_WARN 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: [WRN] BLUESTORE_SLOW_OP_ALERT: 7 OSD(s) experiencing slow operations in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.1 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.2 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.4 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.5 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.8 observed slow operation indications in BlueStore 2024-09-21T05:50:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.10 observed slow operation indications in BlueStore 2024-09-21T05:50:00.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:00 smithi129 ceph-mon[34885]: osd.11 observed slow operation indications in BlueStore 2024-09-21T05:50:02.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:02 smithi181 ceph-mon[34819]: pgmap v415: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 44 MiB/s wr, 5.50k op/s 2024-09-21T05:50:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:02 smithi028 ceph-mon[31227]: pgmap v415: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 44 MiB/s wr, 5.50k op/s 2024-09-21T05:50:02.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:02 smithi129 ceph-mon[34885]: pgmap v415: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 44 MiB/s wr, 5.50k op/s 2024-09-21T05:50:04.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:04 smithi181 ceph-mon[34819]: pgmap v416: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 MiB/s wr, 7.42k op/s 2024-09-21T05:50:04.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:04 smithi028 ceph-mon[31227]: pgmap v416: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 MiB/s wr, 7.42k op/s 2024-09-21T05:50:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:04 smithi129 ceph-mon[34885]: pgmap v416: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 MiB/s wr, 7.42k op/s 2024-09-21T05:50:06.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:50:06.494 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:06.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:06 smithi028 ceph-mon[31227]: pgmap v417: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 MiB/s wr, 5.97k op/s 2024-09-21T05:50:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:06 smithi181 ceph-mon[34819]: pgmap v417: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 MiB/s wr, 5.97k op/s 2024-09-21T05:50:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:06 smithi129 ceph-mon[34885]: pgmap v417: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 MiB/s wr, 5.97k op/s 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 -- 172.21.15.28:0/634322093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f44105db0 msgr2=0x7f8f441061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 --2- 172.21.15.28:0/634322093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f44105db0 0x7f8f441061c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009900 tx=0x7f8f3c02a410 comp rx=0 tx=0).stop 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 -- 172.21.15.28:0/634322093 shutdown_connections 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 --2- 172.21.15.28:0/634322093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f44112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 --2- 172.21.15.28:0/634322093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f4410b6f0 0x7f8f4410bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 --2- 172.21.15.28:0/634322093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f44105db0 0x7f8f441061c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 -- 172.21.15.28:0/634322093 >> 172.21.15.28:0/634322093 conn(0x7f8f44101800 msgr2=0x7f8f44103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 -- 172.21.15.28:0/634322093 shutdown_connections 2024-09-21T05:50:07.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.032+0000 7f8f49763640 1 -- 172.21.15.28:0/634322093 wait complete. 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.033+0000 7f8f49763640 1 Processor -- start 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 -- start start 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f44105db0 0x7f8f4419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 0x7f8f4419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f44115d50 con 0x7f8f4410b6f0 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f44115bd0 con 0x7f8f44105db0 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f49763640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f44115ed0 con 0x7f8f4410c040 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f427fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 0x7f8f4419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33368/0 (socket says 172.21.15.28:33368) 2024-09-21T05:50:07.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.034+0000 7f8f437fe640 1 -- 172.21.15.28:0/359608123 learned_addr learned my addr 172.21.15.28:0/359608123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f437fe640 1 -- 172.21.15.28:0/359608123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f44105db0 msgr2=0x7f8f4419c6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f437fe640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f44105db0 0x7f8f4419c6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f437fe640 1 -- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 msgr2=0x7f8f4419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f437fe640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 0x7f8f4419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f437fe640 1 -- 172.21.15.28:0/359608123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f441a2630 con 0x7f8f4410c040 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.035+0000 7f8f427fc640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 0x7f8f4419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.036+0000 7f8f437fe640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400eb10 tx=0x7f8f3400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.036+0000 7f8f2bfff640 1 -- 172.21.15.28:0/359608123 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34004b20 con 0x7f8f4410c040 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.036+0000 7f8f2bfff640 1 -- 172.21.15.28:0/359608123 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f34004cc0 con 0x7f8f4410c040 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.036+0000 7f8f2bfff640 1 -- 172.21.15.28:0/359608123 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34010670 con 0x7f8f4410c040 2024-09-21T05:50:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.037+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f441a2920 con 0x7f8f4410c040 2024-09-21T05:50:07.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.037+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f441a9080 con 0x7f8f4410c040 2024-09-21T05:50:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.041+0000 7f8f2bfff640 1 -- 172.21.15.28:0/359608123 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8f34005020 con 0x7f8f4410c040 2024-09-21T05:50:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.041+0000 7f8f4fcd7740 1 Processor -- start 2024-09-21T05:50:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.042+0000 7f8f2bfff640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 0x7f8f2c0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.042+0000 7f8f4fcd7740 1 -- start start 2024-09-21T05:50:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.042+0000 7f8f2bfff640 1 -- 172.21.15.28:0/359608123 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8f340e7d10 con 0x7f8f4410c040 2024-09-21T05:50:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.042+0000 7f8f427fc640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 0x7f8f2c0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc17441b0 0x55dfc1b14b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc174c390 0x55dfc1b150d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:33370/0 (socket says 172.21.15.28:33370) 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f42ffd640 1 -- 172.21.15.28:0/4141215301 learned_addr learned my addr 172.21.15.28:0/4141215301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f427fc640 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc17441b0 0x55dfc1b14b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfc1a8ae50 con 0x55dfc174c390 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dfc1a7a280 con 0x55dfc17441b0 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.043+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dfc1a7e090 con 0x55dfc176d070 2024-09-21T05:50:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f437fe640 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc174c390 0x55dfc1b150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 -- 172.21.15.28:0/4141215301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc17441b0 msgr2=0x55dfc1b14b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc17441b0 0x55dfc1b14b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 -- 172.21.15.28:0/4141215301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc174c390 msgr2=0x55dfc1b150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc174c390 0x55dfc1b150d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 -- 172.21.15.28:0/4141215301 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfc17d8650 con 0x55dfc176d070 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f427fc640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 0x7f8f2c0b8c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8f4410b060 tx=0x7f8f38011a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c031440 tx=0x7f8f3c009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60a15b49805cab1c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f297fa640 1 -- 172.21.15.28:0/4141215301 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c03e030 con 0x55dfc176d070 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f297fa640 1 -- 172.21.15.28:0/4141215301 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f3c005020 con 0x55dfc176d070 2024-09-21T05:50:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.044+0000 7f8f297fa640 1 -- 172.21.15.28:0/4141215301 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c035040 con 0x55dfc176d070 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.045+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 msgr2=0x55dfc1b14650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.045+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c031440 tx=0x7f8f3c009620 comp rx=0 tx=0).stop 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 shutdown_connections 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc174c390 0x55dfc1b150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc17441b0 0x55dfc1b14b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/4141215301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b14650 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 >> 172.21.15.28:0/4141215301 conn(0x55dfc1761460 msgr2=0x55dfc1b12540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 shutdown_connections 2024-09-21T05:50:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/4141215301 wait complete. 2024-09-21T05:50:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 -- start start 2024-09-21T05:50:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.046+0000 7f8f4fcd7740 1 Processor -- start 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b28da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b29380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b2e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfc18db700 con 0x55dfc17441b0 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dfc1a15110 con 0x55dfc174c390 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dfc1b22dd0 con 0x55dfc176d070 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f4fcd7740 10 client.0 _ll_register_callbacks cb 0x55dfc1a2e990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b28da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f427fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b29380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b28da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47588/0 (socket says 172.21.15.28:47588) 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f42ffd640 1 -- 172.21.15.28:0/3273389019 learned_addr learned my addr 172.21.15.28:0/3273389019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b2e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 -- 172.21.15.28:0/3273389019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 msgr2=0x55dfc1b29380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b29380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 -- 172.21.15.28:0/3273389019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 msgr2=0x55dfc1b28da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b28da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f437fe640 1 -- 172.21.15.28:0/3273389019 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfc1a42c40 con 0x55dfc176d070 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.047+0000 7f8f427fc640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b29380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.048+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b2e3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f340e5420 tx=0x7f8f340e5950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2123bcf45e061fd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.048+0000 7f8f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.048+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f340b0e60 con 0x55dfc176d070 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.048+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8f340e5b70 con 0x55dfc176d070 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.048+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f3c005020 con 0x55dfc176d070 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.049+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8f340a9290 con 0x55dfc176d070 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.049+0000 7f8f4fcd7740 10 client.26047 fetch_fsmap learned FSMap version 34 2024-09-21T05:50:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.049+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55dfc1b18990 con 0x55dfc176d070 2024-09-21T05:50:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.049+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f3c005020 con 0x55dfc176d070 2024-09-21T05:50:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.049+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f8f34097020 con 0x55dfc176d070 2024-09-21T05:50:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 10 client.26047 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:50:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 10 client.26047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 20 client.26047 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc172b9f0 0x55dfc1b1bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 4 client.26047 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.050+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfc1a8cda0 con 0x55dfc172b9f0 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.051+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc172b9f0 0x55dfc1b1bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.052+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc172b9f0 0x55dfc1b1bf90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ebe841180a7d4ce server_cookie=fb454fe119ed4f90 in_seq=0 out_seq=0 2024-09-21T05:50:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.052+0000 7f8f26ff5640 10 client.26047 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3273389019 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dfc1a8cda0 con 0x55dfc172b9f0 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f26ff5640 10 client.26047 handle_command_reply: tid=0 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 client.26047 shutdown 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 msgr2=0x55dfc1b2e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b2e3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f340e5420 tx=0x7f8f340e5950 comp rx=0 tx=0).stop 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 shutdown_connections 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc172b9f0 0x55dfc1b1bf90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f26ff5640 0 client.26047 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc176d070 0x55dfc1b2e3a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b29380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.053+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3273389019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b28da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.054+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 >> 172.21.15.28:0/3273389019 conn(0x55dfc1761460 msgr2=0x55dfc1b177e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.054+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 shutdown_connections 2024-09-21T05:50:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.054+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3273389019 wait complete. 2024-09-21T05:50:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.054+0000 7f8f4fcd7740 20 client.26047 trim_cache size 0 max 16384 2024-09-21T05:50:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.075+0000 7f8f4fcd7740 1 Processor -- start 2024-09-21T05:50:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.075+0000 7f8f4fcd7740 1 -- start start 2024-09-21T05:50:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.075+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc172b9f0 0x55dfc1b25fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b17750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441cf580 con 0x55dfc17441b0 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441d1210 con 0x55dfc174c390 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441cfc50 con 0x55dfc172b9f0 2024-09-21T05:50:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f427fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f427fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47592/0 (socket says 172.21.15.28:47592) 2024-09-21T05:50:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f427fc640 1 -- 172.21.15.28:0/2275302301 learned_addr learned my addr 172.21.15.28:0/2275302301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.076+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc172b9f0 0x55dfc1b25fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 -- 172.21.15.28:0/2275302301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc172b9f0 msgr2=0x55dfc1b25fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc172b9f0 0x55dfc1b25fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 -- 172.21.15.28:0/2275302301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 msgr2=0x55dfc1b17750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b17750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 -- 172.21.15.28:0/2275302301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfc1b22d10 con 0x55dfc17441b0 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f427fc640 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800fdb0 tx=0x7f8f3801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7c90d6302105f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f28ff9640 1 -- 172.21.15.28:0/2275302301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3801edb0 con 0x55dfc17441b0 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f28ff9640 1 -- 172.21.15.28:0/2275302301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f380035f0 con 0x55dfc17441b0 2024-09-21T05:50:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.077+0000 7f8f28ff9640 1 -- 172.21.15.28:0/2275302301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38028480 con 0x55dfc17441b0 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/2275302301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 msgr2=0x55dfc1b264e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800fdb0 tx=0x7f8f3801c000 comp rx=0 tx=0).stop 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/2275302301 shutdown_connections 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc174c390 0x55dfc1b17750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b264e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/2275302301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc172b9f0 0x55dfc1b25fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/2275302301 >> 172.21.15.28:0/2275302301 conn(0x55dfc1761460 msgr2=0x55dfc1b27420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/2275302301 shutdown_connections 2024-09-21T05:50:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/2275302301 wait complete. 2024-09-21T05:50:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.079+0000 7f8f4fcd7740 1 -- start start 2024-09-21T05:50:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 Processor -- start 2024-09-21T05:50:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 0x55dfc1b379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b37ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 0x55dfc1b9b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441d0a80 con 0x55dfc17441b0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441d0380 con 0x55dfc172b9f0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441cd560 con 0x55dfc174c390 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f4fcd7740 10 client.0 _ll_register_callbacks cb 0x7f8f340a9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 0x55dfc1b379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 0x55dfc1b379a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33826/0 (socket says 172.21.15.28:33826) 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f42ffd640 1 -- 172.21.15.28:0/3708693156 learned_addr learned my addr 172.21.15.28:0/3708693156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 0x55dfc1b9b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.080+0000 7f8f427fc640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b37ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 -- 172.21.15.28:0/3708693156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 msgr2=0x55dfc1b379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 0x55dfc1b379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 -- 172.21.15.28:0/3708693156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 msgr2=0x55dfc1b37ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b37ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 -- 172.21.15.28:0/3708693156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfc1b9ba30 con 0x55dfc174c390 2024-09-21T05:50:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f437fe640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 0x55dfc1b9b4f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8f340e5420 tx=0x7f8f340a9a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3d3d908a34f7fba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f340ca070 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8f340acd80 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f380035f0 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8f340ad630 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f4fcd7740 10 client.26053 fetch_fsmap learned FSMap version 34 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55dfc1b9cc00 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.081+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f380035f0 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f8f3408f020 con 0x55dfc174c390 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 10 client.26053 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 10 client.26053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 20 client.26053 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc176d070 0x55dfc1b9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 4 client.26053 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:50:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfc1a88850 con 0x55dfc176d070 2024-09-21T05:50:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.082+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc176d070 0x55dfc1b9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f42ffd640 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc176d070 0x55dfc1b9f460 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58d95fcaeb13dad server_cookie=8b6d7b2053a99375 in_seq=0 out_seq=0 2024-09-21T05:50:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f26ff5640 10 client.26053 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f26ff5640 1 -- 172.21.15.28:0/3708693156 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1990 (crc 0 0 0) 0x55dfc1a88850 con 0x55dfc176d070 2024-09-21T05:50:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f26ff5640 10 client.26053 handle_command_reply: tid=0 2024-09-21T05:50:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 client.26053 shutdown 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 msgr2=0x55dfc1b9b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 0x55dfc1b9b4f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8f340e5420 tx=0x7f8f340a9a30 comp rx=0 tx=0).stop 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 shutdown_connections 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55dfc176d070 0x55dfc1b9f460 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f26ff5640 0 client.26053 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dfc174c390 0x55dfc1b9b4f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfc17441b0 0x55dfc1b37ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:50:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "45ebe5b7-9e03-4b06-b4b1-cee561706b06", 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:07.094 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.083+0000 7f8f4fcd7740 1 --2- 172.21.15.28:0/3708693156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dfc172b9f0 0x55dfc1b379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.084+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 >> 172.21.15.28:0/3708693156 conn(0x55dfc1761460 msgr2=0x55dfc1b25fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.084+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 shutdown_connections 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.084+0000 7f8f4fcd7740 1 -- 172.21.15.28:0/3708693156 wait complete. 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.084+0000 7f8f4fcd7740 20 client.26053 trim_cache size 0 max 16384 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.085+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 msgr2=0x7f8f2c0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.085+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 0x7f8f2c0b8c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8f4410b060 tx=0x7f8f38011a20 comp rx=0 tx=0).stop 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.085+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 msgr2=0x7f8f441a1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.085+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400eb10 tx=0x7f8f3400c2d0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 shutdown_connections 2024-09-21T05:50:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f4410c040 0x7f8f441a1f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8f2c0b6780 0x7f8f2c0b8c90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f4410b6f0 0x7f8f4419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 --2- 172.21.15.28:0/359608123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f44105db0 0x7f8f4419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 >> 172.21.15.28:0/359608123 conn(0x7f8f44101800 msgr2=0x7f8f44103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 shutdown_connections 2024-09-21T05:50:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:07.086+0000 7f8f49763640 1 -- 172.21.15.28:0/359608123 wait complete. 2024-09-21T05:50:07.931 INFO:tasks.cephfs.filesystem:scrub status for tag:45ebe5b7-9e03-4b06-b4b1-cee561706b06 - {'path': '/', 'tag': '45ebe5b7-9e03-4b06-b4b1-cee561706b06', 'options': 'recursive,force'} 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f7135ffb640 1 -- 172.21.15.28:0/2326254713 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c3516a0f0 con 0x559c34de61c0 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f7135ffb640 10 client.16416 handle_command_reply: tid=0 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 client.16416 shutdown 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 msgr2=0x559c35281eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7148003d80 tx=0x7f714802ef90 comp rx=0 tx=0).stop 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 shutdown_connections 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x559c34de61c0 0x559c35285ca0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f7135ffb640 0 client.16416 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c34e521b0 0x559c35281eb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559c34e314d0 0x559c3520d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 --2- 172.21.15.28:0/2326254713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559c34e292f0 0x559c3520cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 >> 172.21.15.28:0/2326254713 conn(0x559c34e46300 msgr2=0x559c352020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 shutdown_connections 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.689+0000 7f715be83740 1 -- 172.21.15.28:0/2326254713 wait complete. 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.690+0000 7f715be83740 20 client.16416 trim_cache size 0 max 16384 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:50:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 msgr2=0x7f71380b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 0x7f71380b8c10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7140105220 tx=0x7f714003b4f0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 msgr2=0x7f715019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7140005940 tx=0x7f71400040c0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 shutdown_connections 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f715010dd70 0x7f715019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71380b6700 0x7f71380b8c10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71501051f0 0x7f715019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 --2- 172.21.15.28:0/1631567844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71501047c0 0x7f715019a490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 >> 172.21.15.28:0/1631567844 conn(0x7f715007a6f0 msgr2=0x7f7150100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 shutdown_connections 2024-09-21T05:50:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:08.691+0000 7f7154f19640 1 -- 172.21.15.28:0/1631567844 wait complete. 2024-09-21T05:50:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:08 smithi181 ceph-mon[34819]: pgmap v418: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 58 MiB/s wr, 6.87k op/s 2024-09-21T05:50:08.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:08 smithi028 ceph-mon[31227]: pgmap v418: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 58 MiB/s wr, 6.87k op/s 2024-09-21T05:50:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:08 smithi129 ceph-mon[34885]: pgmap v418: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 58 MiB/s wr, 6.87k op/s 2024-09-21T05:50:10.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:10 smithi181 ceph-mon[34819]: pgmap v419: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 MiB/s wr, 6.35k op/s 2024-09-21T05:50:10.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:10 smithi028 ceph-mon[31227]: pgmap v419: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 MiB/s wr, 6.35k op/s 2024-09-21T05:50:10.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:10 smithi129 ceph-mon[34885]: pgmap v419: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 MiB/s wr, 6.35k op/s 2024-09-21T05:50:11.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:11 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:50:11.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:11 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:50:11.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:11 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:50:11.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:11 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:50:11.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:11 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:50:11.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:11 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:50:12.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:11 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:50:12.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:11 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:50:12.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:11 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:50:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:12 smithi028 ceph-mon[31227]: pgmap v420: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 MiB/s wr, 5.81k op/s 2024-09-21T05:50:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:12 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:50:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:12 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:50:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:12 smithi129 ceph-mon[34885]: pgmap v420: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 MiB/s wr, 5.81k op/s 2024-09-21T05:50:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:12 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:50:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:12 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:50:13.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:12 smithi181 ceph-mon[34819]: pgmap v420: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 MiB/s wr, 5.81k op/s 2024-09-21T05:50:13.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:12 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:50:13.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:12 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:50:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:14 smithi028 ceph-mon[31227]: pgmap v421: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 67 MiB/s wr, 7.76k op/s 2024-09-21T05:50:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:14 smithi129 ceph-mon[34885]: pgmap v421: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 67 MiB/s wr, 7.76k op/s 2024-09-21T05:50:15.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:14 smithi181 ceph-mon[34819]: pgmap v421: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 67 MiB/s wr, 7.76k op/s 2024-09-21T05:50:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:16 smithi028 ceph-mon[31227]: pgmap v422: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 6.15k op/s 2024-09-21T05:50:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:16 smithi129 ceph-mon[34885]: pgmap v422: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 6.15k op/s 2024-09-21T05:50:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:16 smithi181 ceph-mon[34819]: pgmap v422: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 6.15k op/s 2024-09-21T05:50:18.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:18 smithi028 ceph-mon[31227]: pgmap v423: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 59 MiB/s wr, 7.02k op/s 2024-09-21T05:50:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:18 smithi129 ceph-mon[34885]: pgmap v423: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 59 MiB/s wr, 7.02k op/s 2024-09-21T05:50:19.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:18 smithi181 ceph-mon[34819]: pgmap v423: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 59 MiB/s wr, 7.02k op/s 2024-09-21T05:50:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:20 smithi028 ceph-mon[31227]: pgmap v424: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 MiB/s wr, 6.82k op/s 2024-09-21T05:50:20.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:20 smithi129 ceph-mon[34885]: pgmap v424: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 MiB/s wr, 6.82k op/s 2024-09-21T05:50:21.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:20 smithi181 ceph-mon[34819]: pgmap v424: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 MiB/s wr, 6.82k op/s 2024-09-21T05:50:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 -- 172.21.15.28:0/3595066092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 msgr2=0x7f47141105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 --2- 172.21.15.28:0/3595066092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f47141105b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f470c009ca0 tx=0x7f470c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 -- 172.21.15.28:0/3595066092 shutdown_connections 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 --2- 172.21.15.28:0/3595066092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f47141105b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 --2- 172.21.15.28:0/3595066092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714109430 0x7f4714109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 --2- 172.21.15.28:0/3595066092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714102060 0x7f4714102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 -- 172.21.15.28:0/3595066092 >> 172.21.15.28:0/3595066092 conn(0x7f4714079e90 msgr2=0x7f471407a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 -- 172.21.15.28:0/3595066092 shutdown_connections 2024-09-21T05:50:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.829+0000 7f471adff640 1 -- 172.21.15.28:0/3595066092 wait complete. 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.830+0000 7f471adff640 1 Processor -- start 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.830+0000 7f471adff640 1 -- start start 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 0x7f471419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 0x7f471419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4714113a40 con 0x7f4714109d80 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47141138c0 con 0x7f4714102060 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471adff640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4714113bc0 con 0x7f4714109430 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 0x7f471419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471939c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471939c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52266/0 (socket says 172.21.15.28:52266) 2024-09-21T05:50:21.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.831+0000 7f471939c640 1 -- 172.21.15.28:0/1579855791 learned_addr learned my addr 172.21.15.28:0/1579855791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4713fff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 0x7f471419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 -- 172.21.15.28:0/1579855791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 msgr2=0x7f471419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 0x7f471419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 -- 172.21.15.28:0/1579855791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 msgr2=0x7f471419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 0x7f471419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 -- 172.21.15.28:0/1579855791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47141a0350 con 0x7f4714109d80 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4713fff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 0x7f471419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4718b9b640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 0x7f471419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471939c640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f470c005cc0 tx=0x7f470c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4711ffb640 1 -- 172.21.15.28:0/1579855791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c003b70 con 0x7f4714109d80 2024-09-21T05:50:21.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47141a05e0 con 0x7f4714109d80 2024-09-21T05:50:21.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4711ffb640 1 -- 172.21.15.28:0/1579855791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c0318f0 con 0x7f4714109d80 2024-09-21T05:50:21.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.832+0000 7f4711ffb640 1 -- 172.21.15.28:0/1579855791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c02f950 con 0x7f4714109d80 2024-09-21T05:50:21.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.833+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47141a6dd0 con 0x7f4714109d80 2024-09-21T05:50:21.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 Processor -- start 2024-09-21T05:50:21.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4711ffb640 1 -- 172.21.15.28:0/1579855791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f470c02fb50 con 0x7f4714109d80 2024-09-21T05:50:21.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 -- start start 2024-09-21T05:50:21.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d7709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d770ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d771430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d3d6e56f0 con 0x555d3d3a22f0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555d3d6d4b20 con 0x555d3d3aa4d0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.836+0000 7f4721e47740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555d3d6d8930 con 0x555d3d3cb1b0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d770ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d770ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52272/0 (socket says 172.21.15.28:52272) 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d771430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4718b9b640 1 -- 172.21.15.28:0/3598067169 learned_addr learned my addr 172.21.15.28:0/3598067169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d7709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 -- 172.21.15.28:0/3598067169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 msgr2=0x555d3d7709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d7709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 -- 172.21.15.28:0/3598067169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 msgr2=0x555d3d770ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d770ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 -- 172.21.15.28:0/3598067169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3d69a530 con 0x555d3d3aa4d0 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4711ffb640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 0x7f47040b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4713fff640 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d771430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4714075930 tx=0x7f47000095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64115aa1ce668612 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.837+0000 7f4718b9b640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 0x7f47040b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.838+0000 7f4711ffb640 1 -- 172.21.15.28:0/1579855791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f470c108910 con 0x7f4714109d80 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.838+0000 7f46feffd640 1 -- 172.21.15.28:0/3598067169 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47000142d0 con 0x555d3d3aa4d0 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.838+0000 7f46feffd640 1 -- 172.21.15.28:0/3598067169 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470000f760 con 0x555d3d3aa4d0 2024-09-21T05:50:21.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.838+0000 7f46feffd640 1 -- 172.21.15.28:0/3598067169 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47000044e0 con 0x555d3d3aa4d0 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- 172.21.15.28:0/3598067169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 msgr2=0x555d3d771430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4718b9b640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 0x7f47040b8b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x555d3d76f630 tx=0x7f4708005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d771430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4714075930 tx=0x7f47000095d0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- 172.21.15.28:0/3598067169 shutdown_connections 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d771430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d770ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 --2- 172.21.15.28:0/3598067169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d7709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- 172.21.15.28:0/3598067169 >> 172.21.15.28:0/3598067169 conn(0x555d3d3bf300 msgr2=0x555d3d76e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- 172.21.15.28:0/3598067169 shutdown_connections 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- 172.21.15.28:0/3598067169 wait complete. 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 -- start start 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.839+0000 7f4721e47740 1 Processor -- start 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d77c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d78c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d3d60cab0 con 0x555d3d3aa4d0 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555d3d67a730 con 0x555d3d3a22f0 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555d3d7834e0 con 0x555d3d3cb1b0 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4721e47740 10 client.0 _ll_register_callbacks cb 0x555d3d6ce6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d77c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4713fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4713fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52286/0 (socket says 172.21.15.28:52286) 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f4713fff640 1 -- 172.21.15.28:0/1050537806 learned_addr learned my addr 172.21.15.28:0/1050537806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.840+0000 7f471939c640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d78c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 -- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 msgr2=0x555d3d78c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d78c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 -- 172.21.15.28:0/1050537806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 msgr2=0x555d3d77c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d77c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 -- 172.21.15.28:0/1050537806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3d436790 con 0x555d3d3aa4d0 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4713fff640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4700009570 tx=0x7f4700014380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60a6a73b37f8d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f46f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700014510 con 0x555d3d3aa4d0 2024-09-21T05:50:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4700004050 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.841+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f470000f760 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f47000042b0 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 10 client.16464 fetch_fsmap learned FSMap version 34 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555d3d4af620 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470000f760 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f4700028470 con 0x555d3d3aa4d0 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 10 client.16464 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 20 client.16464 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d389d80 0x555d3d792470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 4 client.16464 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.842+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d3d6e7640 con 0x555d3d389d80 2024-09-21T05:50:21.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.843+0000 7f4718b9b640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d389d80 0x555d3d792470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.843+0000 7f46f3fff640 10 client.16464 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.843+0000 7f4718b9b640 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d389d80 0x555d3d792470 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9866eae0b7f53b8 server_cookie=d43c02370e7e3316 in_seq=0 out_seq=0 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.844+0000 7f46f3fff640 1 -- 172.21.15.28:0/1050537806 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555d3d6e7640 con 0x555d3d389d80 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.844+0000 7f46f3fff640 10 client.16464 handle_command_reply: tid=0 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.844+0000 7f4721e47740 1 client.16464 shutdown 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.844+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 msgr2=0x555d3d77c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.844+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4700009570 tx=0x7f4700014380 comp rx=0 tx=0).stop 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 shutdown_connections 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d389d80 0x555d3d792470 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d78c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3aa4d0 0x555d3d77c5b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 --2- 172.21.15.28:0/1050537806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d77c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 >> 172.21.15.28:0/1050537806 conn(0x555d3d3bf300 msgr2=0x555d3d7719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f46f3fff640 0 client.16464 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 shutdown_connections 2024-09-21T05:50:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 1 -- 172.21.15.28:0/1050537806 wait complete. 2024-09-21T05:50:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.845+0000 7f4721e47740 20 client.16464 trim_cache size 0 max 16384 2024-09-21T05:50:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 Processor -- start 2024-09-21T05:50:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 -- start start 2024-09-21T05:50:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d779240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d779780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47141ad450 con 0x555d3d3a22f0 2024-09-21T05:50:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47141af0e0 con 0x555d3d3aa4d0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4721e47740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47141adb20 con 0x555d3d3cb1b0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52302/0 (socket says 172.21.15.28:52302) 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4718b9b640 1 -- 172.21.15.28:0/3870235417 learned_addr learned my addr 172.21.15.28:0/3870235417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.865+0000 7f4713fff640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d779240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f471939c640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d779780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 -- 172.21.15.28:0/3870235417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 msgr2=0x555d3d779780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d779780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 -- 172.21.15.28:0/3870235417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 msgr2=0x555d3d77df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 -- 172.21.15.28:0/3870235417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3d77a550 con 0x555d3d3aa4d0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4718b9b640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f4713fff640 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d779240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4700009b00 tx=0x7f47000097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8081f623a687a01f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.866+0000 7f46fe7fc640 1 -- 172.21.15.28:0/3870235417 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700035070 con 0x555d3d3aa4d0 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.867+0000 7f46fe7fc640 1 -- 172.21.15.28:0/3870235417 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4700023070 con 0x555d3d3aa4d0 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.867+0000 7f46fe7fc640 1 -- 172.21.15.28:0/3870235417 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470000fb20 con 0x555d3d3aa4d0 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 -- 172.21.15.28:0/3870235417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 msgr2=0x555d3d779240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d779240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4700009b00 tx=0x7f47000097f0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 -- 172.21.15.28:0/3870235417 shutdown_connections 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3cb1b0 0x555d3d779780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3aa4d0 0x555d3d779240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 --2- 172.21.15.28:0/3870235417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3a22f0 0x555d3d77df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.868+0000 7f4721e47740 1 -- 172.21.15.28:0/3870235417 >> 172.21.15.28:0/3870235417 conn(0x555d3d3bf300 msgr2=0x555d3d78b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.869+0000 7f4721e47740 1 -- 172.21.15.28:0/3870235417 shutdown_connections 2024-09-21T05:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.869+0000 7f4721e47740 1 -- 172.21.15.28:0/3870235417 wait complete. 2024-09-21T05:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.869+0000 7f4721e47740 1 -- start start 2024-09-21T05:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.869+0000 7f4721e47740 1 Processor -- start 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 0x555d3d795620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 0x555d3d7f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47141ae950 con 0x555d3d3cb1b0 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47141ae250 con 0x555d3d3a22f0 2024-09-21T05:50:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47141ab430 con 0x555d3d3aa4d0 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4721e47740 10 client.0 _ll_register_callbacks cb 0x7f4708009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f471939c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 0x555d3d7f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4713fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 0x555d3d795620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:51946/0 (socket says 172.21.15.28:51946) 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f471939c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 0x555d3d7f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52308/0 (socket says 172.21.15.28:52308) 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 -- 172.21.15.28:0/307581125 learned_addr learned my addr 172.21.15.28:0/307581125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4713fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 0x555d3d795620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:43162/0 (socket says 172.21.15.28:43162) 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 -- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 msgr2=0x555d3d795620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 0x555d3d795620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 -- 172.21.15.28:0/307581125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 msgr2=0x555d3d7f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 0x555d3d7f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 -- 172.21.15.28:0/307581125 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3d7f3240 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f4718b9b640 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4708004650 tx=0x7f4708017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26af786f795a89b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.870+0000 7f46f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708036060 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4708031440 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4700023070 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4708031640 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 10 client.26036 fetch_fsmap learned FSMap version 34 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555d3d7fc8b0 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4700023070 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 34) ==== 8394+0+0 (secure 0 0 0) 0x7f470801f7c0 con 0x555d3d3a22f0 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 10 client.26036 fetch_fsmap finished waiting for FSMap version 34 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 10 client.26036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 20 client.26036 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d35f1c0 0x555d3d7ff190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 4 client.26036 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d3d6e30f0 con 0x555d3d35f1c0 2024-09-21T05:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.871+0000 7f4713fff640 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d35f1c0 0x555d3d7ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.872+0000 7f46f3fff640 10 client.26036 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:21.872+0000 7f4713fff640 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d35f1c0 0x555d3d7ff190 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f632d406ad34adb3 server_cookie=b19fc2e898ee1418 in_seq=0 out_seq=0 2024-09-21T05:50:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:22 smithi028 ceph-mon[31227]: pgmap v425: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 47 MiB/s wr, 5.79k op/s 2024-09-21T05:50:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:22 smithi129 ceph-mon[34885]: pgmap v425: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 47 MiB/s wr, 5.79k op/s 2024-09-21T05:50:23.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:22 smithi181 ceph-mon[34819]: pgmap v425: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 47 MiB/s wr, 5.79k op/s 2024-09-21T05:50:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:24 smithi028 ceph-mon[31227]: pgmap v426: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 57 MiB/s wr, 7.05k op/s 2024-09-21T05:50:24.845 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:24 smithi181 ceph-mon[34819]: pgmap v426: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 57 MiB/s wr, 7.05k op/s 2024-09-21T05:50:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:24 smithi129 ceph-mon[34885]: pgmap v426: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 57 MiB/s wr, 7.05k op/s 2024-09-21T05:50:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:26 smithi028 ceph-mon[31227]: pgmap v427: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 42 MiB/s wr, 5.26k op/s 2024-09-21T05:50:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:26 smithi129 ceph-mon[34885]: pgmap v427: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 42 MiB/s wr, 5.26k op/s 2024-09-21T05:50:27.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:26 smithi181 ceph-mon[34819]: pgmap v427: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 42 MiB/s wr, 5.26k op/s 2024-09-21T05:50:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:28 smithi129 ceph-mon[34885]: pgmap v428: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 52 MiB/s wr, 6.43k op/s 2024-09-21T05:50:29.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:28 smithi181 ceph-mon[34819]: pgmap v428: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 52 MiB/s wr, 6.43k op/s 2024-09-21T05:50:29.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:28 smithi028 ceph-mon[31227]: pgmap v428: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 52 MiB/s wr, 6.43k op/s 2024-09-21T05:50:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:30 smithi129 ceph-mon[34885]: pgmap v429: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 46 MiB/s wr, 5.84k op/s 2024-09-21T05:50:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:30 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:31.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:30 smithi181 ceph-mon[34819]: pgmap v429: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 46 MiB/s wr, 5.84k op/s 2024-09-21T05:50:31.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:30 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:31.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:30 smithi028 ceph-mon[31227]: pgmap v429: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 46 MiB/s wr, 5.84k op/s 2024-09-21T05:50:31.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:30 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:31.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:31 smithi129 ceph-mon[34885]: pgmap v430: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 4.66k op/s 2024-09-21T05:50:32.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:31 smithi181 ceph-mon[34819]: pgmap v430: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 4.66k op/s 2024-09-21T05:50:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:31 smithi028 ceph-mon[31227]: pgmap v430: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 4.66k op/s 2024-09-21T05:50:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.794+0000 7f46f3fff640 1 -- 172.21.15.28:0/307581125 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d3d6e30f0 con 0x555d3d35f1c0 2024-09-21T05:50:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.794+0000 7f46f3fff640 10 client.26036 handle_command_reply: tid=0 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.794+0000 7f4721e47740 1 client.26036 shutdown 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 msgr2=0x555d3d795040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4708004650 tx=0x7f4708017000 comp rx=0 tx=0).stop 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 shutdown_connections 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555d3d35f1c0 0x555d3d7ff190 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d3d3cb1b0 0x555d3d7f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555d3d3aa4d0 0x555d3d795620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f46f3fff640 0 client.26036 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 --2- 172.21.15.28:0/307581125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d3d3a22f0 0x555d3d795040 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 >> 172.21.15.28:0/307581125 conn(0x555d3d3bf300 msgr2=0x555d3d77dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 shutdown_connections 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 1 -- 172.21.15.28:0/307581125 wait complete. 2024-09-21T05:50:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.795+0000 7f4721e47740 20 client.26036 trim_cache size 0 max 16384 2024-09-21T05:50:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:50:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:50:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:50:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 msgr2=0x7f47040b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 0x7f47040b8b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x555d3d76f630 tx=0x7f4708005f90 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 msgr2=0x7f471419fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f470c005cc0 tx=0x7f470c031700 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 shutdown_connections 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f47040b6680 0x7f47040b8b90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714109d80 0x7f471419fc50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4714109430 0x7f471419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 --2- 172.21.15.28:0/1579855791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4714102060 0x7f471419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 >> 172.21.15.28:0/1579855791 conn(0x7f4714079e90 msgr2=0x7f4714107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 shutdown_connections 2024-09-21T05:50:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:32.796+0000 7f471adff640 1 -- 172.21.15.28:0/1579855791 wait complete. 2024-09-21T05:50:34.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:33 smithi181 ceph-mon[34819]: pgmap v431: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 MiB/s wr, 6.69k op/s 2024-09-21T05:50:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:33 smithi028 ceph-mon[31227]: pgmap v431: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 MiB/s wr, 6.69k op/s 2024-09-21T05:50:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:33 smithi129 ceph-mon[34885]: pgmap v431: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 MiB/s wr, 6.69k op/s 2024-09-21T05:50:36.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:35 smithi181 ceph-mon[34819]: pgmap v432: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 MiB/s wr, 5.62k op/s 2024-09-21T05:50:36.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:35 smithi028 ceph-mon[31227]: pgmap v432: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 MiB/s wr, 5.62k op/s 2024-09-21T05:50:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:35 smithi129 ceph-mon[34885]: pgmap v432: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 MiB/s wr, 5.62k op/s 2024-09-21T05:50:37.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:50:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:38.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:37 smithi181 ceph-mon[34819]: pgmap v433: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 60 MiB/s wr, 7.67k op/s 2024-09-21T05:50:38.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:37 smithi028 ceph-mon[31227]: pgmap v433: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 60 MiB/s wr, 7.67k op/s 2024-09-21T05:50:38.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:37 smithi129 ceph-mon[34885]: pgmap v433: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 60 MiB/s wr, 7.67k op/s 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 -- 172.21.15.28:0/2577345343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb710102a60 msgr2=0x7fb710102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 --2- 172.21.15.28:0/2577345343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb710102a60 0x7fb710102e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb700009860 tx=0x7fb70002a340 comp rx=0 tx=0).stop 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 -- 172.21.15.28:0/2577345343 shutdown_connections 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 --2- 172.21.15.28:0/2577345343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 0x7fb71010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 --2- 172.21.15.28:0/2577345343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7101083a0 0x7fb7101087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 --2- 172.21.15.28:0/2577345343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb710102a60 0x7fb710102e70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 -- 172.21.15.28:0/2577345343 >> 172.21.15.28:0/2577345343 conn(0x7fb710079730 msgr2=0x7fb710079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 -- 172.21.15.28:0/2577345343 shutdown_connections 2024-09-21T05:50:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.779+0000 7fb71606c640 1 -- 172.21.15.28:0/2577345343 wait complete. 2024-09-21T05:50:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.780+0000 7fb71606c640 1 Processor -- start 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.780+0000 7fb71606c640 1 -- start start 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb710102a60 0x7fb71019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 0x7fb71019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb71006bab0 con 0x7fb710102a60 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb71006b930 con 0x7fb7101083a0 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb71606c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb71006bc30 con 0x7fb710108cf0 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60934/0 (socket says 172.21.15.28:60934) 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 -- 172.21.15.28:0/4233342244 learned_addr learned my addr 172.21.15.28:0/4233342244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70ffff640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 0x7fb71019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb710102a60 0x7fb71019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 -- 172.21.15.28:0/4233342244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 msgr2=0x7fb71019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 0x7fb71019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 -- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb710102a60 msgr2=0x7fb71019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb710102a60 0x7fb71019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.781+0000 7fb70effd640 1 -- 172.21.15.28:0/4233342244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7101a03a0 con 0x7fb7101083a0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb70effd640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb70400e9a0 tx=0x7fb70400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb70cff9640 1 -- 172.21.15.28:0/4233342244 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70400cd80 con 0x7fb7101083a0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7101a0690 con 0x7fb7101083a0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb70cff9640 1 -- 172.21.15.28:0/4233342244 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7040035f0 con 0x7fb7101083a0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb70cff9640 1 -- 172.21.15.28:0/4233342244 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704010690 con 0x7fb7101083a0 2024-09-21T05:50:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.782+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7101a6df0 con 0x7fb7101083a0 2024-09-21T05:50:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 Processor -- start 2024-09-21T05:50:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb70cff9640 1 -- 172.21.15.28:0/4233342244 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb704003ac0 con 0x7fb7101083a0 2024-09-21T05:50:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 -- start start 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054adb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f005423e50 con 0x55f0050dd1b0 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f005413280 con 0x55f005106070 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.786+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f005417090 con 0x55f0050e5390 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60938/0 (socket says 172.21.15.28:60938) 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 -- 172.21.15.28:0/2616125040 learned_addr learned my addr 172.21.15.28:0/2616125040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70effd640 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 -- 172.21.15.28:0/2616125040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 msgr2=0x55f0054ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 -- 172.21.15.28:0/2616125040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 msgr2=0x55f0054adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 -- 172.21.15.28:0/2616125040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f005171650 con 0x55f005106070 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70cff9640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 0x7fb6f80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.787+0000 7fb70ffff640 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb708011440 tx=0x7fb708011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6480cdb8f6ad21d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.788+0000 7fb6f5ffb640 1 -- 172.21.15.28:0/2616125040 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708003c30 con 0x55f005106070 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.788+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 0x7fb6f80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.788+0000 7fb70cff9640 1 -- 172.21.15.28:0/4233342244 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb7040e7b20 con 0x7fb7101083a0 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.788+0000 7fb6f5ffb640 1 -- 172.21.15.28:0/2616125040 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb708003dd0 con 0x55f005106070 2024-09-21T05:50:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.788+0000 7fb6f5ffb640 1 -- 172.21.15.28:0/2616125040 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70800fae0 con 0x55f005106070 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 0x7fb6f80b8c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a930 tx=0x7fb7000309f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 -- 172.21.15.28:0/2616125040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 msgr2=0x55f0054ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb708011440 tx=0x7fb708011900 comp rx=0 tx=0).stop 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 -- 172.21.15.28:0/2616125040 shutdown_connections 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/2616125040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054ad650 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 -- 172.21.15.28:0/2616125040 >> 172.21.15.28:0/2616125040 conn(0x55f0050fa460 msgr2=0x55f0054ab540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 -- 172.21.15.28:0/2616125040 shutdown_connections 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.789+0000 7fb71c5e0740 1 -- 172.21.15.28:0/2616125040 wait complete. 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.790+0000 7fb71c5e0740 1 -- start start 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.790+0000 7fb71c5e0740 1 Processor -- start 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054b8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f005274700 con 0x55f0050dd1b0 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f0053ae110 con 0x55f005106070 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f0054b9de0 con 0x55f0050e5390 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb71c5e0740 10 client.0 _ll_register_callbacks cb 0x55f0053c7990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58010/0 (socket says 172.21.15.28:58010) 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70f7fe640 1 -- 172.21.15.28:0/913026488 learned_addr learned my addr 172.21.15.28:0/913026488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70effd640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 -- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 msgr2=0x55f0054b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 -- 172.21.15.28:0/913026488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 msgr2=0x55f0054b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70ffff640 1 -- 172.21.15.28:0/913026488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0053dbc40 con 0x55f005106070 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.791+0000 7fb70effd640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb70ffff640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054b8630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb708009c30 tx=0x7fb708003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=389cb5088df06b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708028070 con 0x55f005106070 2024-09-21T05:50:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb708003dd0 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb70800fa30 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb708034050 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 10 client.26048 fetch_fsmap learned FSMap version 35 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f0054d02f0 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb708003dd0 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 35) ==== 8390+0+0 (secure 0 0 0) 0x7fb7080046a0 con 0x55f005106070 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 10 client.26048 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 10 client.26048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 20 client.26048 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.792+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f0050c49f0 0x55f0054d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.793+0000 7fb71c5e0740 4 client.26048 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.793+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f005425da0 con 0x55f0050c49f0 2024-09-21T05:50:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.793+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f0050c49f0 0x55f0054d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.793+0000 7fb6eeffd640 10 client.26048 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.794+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f0050c49f0 0x55f0054d36b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156798e3b32f3584 server_cookie=675e68f3b100a329 in_seq=0 out_seq=0 2024-09-21T05:50:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.794+0000 7fb6eeffd640 1 -- 172.21.15.28:0/913026488 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f005425da0 con 0x55f0050c49f0 2024-09-21T05:50:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.794+0000 7fb6eeffd640 10 client.26048 handle_command_reply: tid=0 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 client.26048 shutdown 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 msgr2=0x55f0054b8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054b8630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb708009c30 tx=0x7fb708003d60 comp rx=0 tx=0).stop 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 shutdown_connections 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f0050c49f0 0x55f0054d36b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb6eeffd640 0 client.26048 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f005106070 0x55f0054b8630 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f0054b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/913026488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 >> 172.21.15.28:0/913026488 conn(0x55f0050fa460 msgr2=0x55f0054b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 shutdown_connections 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 1 -- 172.21.15.28:0/913026488 wait complete. 2024-09-21T05:50:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.795+0000 7fb71c5e0740 20 client.26048 trim_cache size 0 max 16384 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 Processor -- start 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 -- start start 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054bc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050e5390 0x55f0054aab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101ad550 con 0x55f0050dd1b0 2024-09-21T05:50:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101af1b0 con 0x55f0050e5390 2024-09-21T05:50:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101adc20 con 0x55f0050c49f0 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49638/0 (socket says 172.21.15.28:49638) 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.814+0000 7fb70f7fe640 1 -- 172.21.15.28:0/943502151 learned_addr learned my addr 172.21.15.28:0/943502151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70effd640 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054bc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70ffff640 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050e5390 0x55f0054aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 -- 172.21.15.28:0/943502151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050e5390 msgr2=0x55f0054aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050e5390 0x55f0054aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 -- 172.21.15.28:0/943502151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 msgr2=0x55f0054bc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054bc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 -- 172.21.15.28:0/943502151 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0054af940 con 0x55f0050c49f0 2024-09-21T05:50:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.815+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a860 tx=0x7fb70003c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b71822ee46dced71 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.816+0000 7fb6f57fa640 1 -- 172.21.15.28:0/943502151 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700056070 con 0x55f0050c49f0 2024-09-21T05:50:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.816+0000 7fb6f57fa640 1 -- 172.21.15.28:0/943502151 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb700038040 con 0x55f0050c49f0 2024-09-21T05:50:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.816+0000 7fb6f57fa640 1 -- 172.21.15.28:0/943502151 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700037040 con 0x55f0050c49f0 2024-09-21T05:50:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 -- 172.21.15.28:0/943502151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 msgr2=0x55f0054b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a860 tx=0x7fb70003c000 comp rx=0 tx=0).stop 2024-09-21T05:50:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 -- 172.21.15.28:0/943502151 shutdown_connections 2024-09-21T05:50:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050e5390 0x55f0054aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050dd1b0 0x55f0054bc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/943502151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050c49f0 0x55f0054b4d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 -- 172.21.15.28:0/943502151 >> 172.21.15.28:0/943502151 conn(0x55f0050fa460 msgr2=0x55f0054b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 -- 172.21.15.28:0/943502151 shutdown_connections 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.817+0000 7fb71c5e0740 1 -- 172.21.15.28:0/943502151 wait complete. 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.818+0000 7fb71c5e0740 1 -- start start 2024-09-21T05:50:38.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.818+0000 7fb71c5e0740 1 Processor -- start 2024-09-21T05:50:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 0x55f0054ca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f005536fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101aea20 con 0x55f0050c49f0 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101ae350 con 0x55f0050dd1b0 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101ab530 con 0x55f0050e5390 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb71c5e0740 10 client.0 _ll_register_callbacks cb 0x7fb700035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 0x55f0054ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58034/0 (socket says 172.21.15.28:58034) 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70f7fe640 1 -- 172.21.15.28:0/1140841942 learned_addr learned my addr 172.21.15.28:0/1140841942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70effd640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 0x55f0054ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60954/0 (socket says 172.21.15.28:60954) 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.819+0000 7fb70ffff640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f005536fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 -- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 msgr2=0x55f005536fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f005536fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 -- 172.21.15.28:0/1140841942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 msgr2=0x55f0054ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 0x55f0054ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 -- 172.21.15.28:0/1140841942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f005537630 con 0x55f0050c49f0 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb70f7fe640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a380 tx=0x7fb700030d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39658a0f0d005382 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb6eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700056070 con 0x55f0050c49f0 2024-09-21T05:50:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb700038040 con 0x55f0050c49f0 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb7000038b0 con 0x55f0050c49f0 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb700037040 con 0x55f0050c49f0 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb71c5e0740 10 client.16485 fetch_fsmap learned FSMap version 35 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.820+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f00553e980 con 0x55f0050c49f0 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb700038040 con 0x55f0050c49f0 2024-09-21T05:50:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8390+0+0 (secure 0 0 0) 0x7fb700050070 con 0x55f0050c49f0 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 10 client.16485 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 20 client.16485 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f005106070 0x55f005541180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 4 client.16485 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f005421850 con 0x55f005106070 2024-09-21T05:50:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.821+0000 7fb70effd640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f005106070 0x55f005541180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.822+0000 7fb6eeffd640 10 client.16485 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb70effd640 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f005106070 0x55f005541180 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d20ca06d3130a2 server_cookie=a38163c9f979eb4c in_seq=0 out_seq=0 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb6eeffd640 1 -- 172.21.15.28:0/1140841942 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1794 (crc 0 0 0) 0x55f005421850 con 0x55f005106070 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb6eeffd640 10 client.16485 handle_command_reply: tid=0 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb71c5e0740 1 client.16485 shutdown 2024-09-21T05:50:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 msgr2=0x55f0054ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.823+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a380 tx=0x7fb700030d20 comp rx=0 tx=0).stop 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 shutdown_connections 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f005106070 0x55f005541180 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0050e5390 0x55f005536fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb6eeffd640 0 client.16485 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0050dd1b0 0x55f0054ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 --2- 172.21.15.28:0/1140841942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0050c49f0 0x55f0054ca180 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 >> 172.21.15.28:0/1140841942 conn(0x55f0050fa460 msgr2=0x55f0054b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 shutdown_connections 2024-09-21T05:50:38.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 1 -- 172.21.15.28:0/1140841942 wait complete. 2024-09-21T05:50:38.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.824+0000 7fb71c5e0740 20 client.16485 trim_cache size 0 max 16384 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:50:38.839 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:38.840 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.842 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.843 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.844 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:38.845 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.825+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 msgr2=0x7fb6f80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.825+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 0x7fb6f80b8c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb70002a930 tx=0x7fb7000309f0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.825+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 msgr2=0x7fb71019a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.825+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb70400e9a0 tx=0x7fb70400eda0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 shutdown_connections 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb710108cf0 0x7fb71019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7101083a0 0x7fb71019a9d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6f80b6700 0x7fb6f80b8c10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 --2- 172.21.15.28:0/4233342244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb710102a60 0x7fb71019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 >> 172.21.15.28:0/4233342244 conn(0x7fb710079730 msgr2=0x7fb7101013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 shutdown_connections 2024-09-21T05:50:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:38.826+0000 7fb71606c640 1 -- 172.21.15.28:0/4233342244 wait complete. 2024-09-21T05:50:39.689 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:50:39.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:50:39.984 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:40.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:39 smithi028 ceph-mon[31227]: pgmap v434: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 MiB/s wr, 6.12k op/s 2024-09-21T05:50:40.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:39 smithi181 ceph-mon[34819]: pgmap v434: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 MiB/s wr, 6.12k op/s 2024-09-21T05:50:40.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:39 smithi129 ceph-mon[34885]: pgmap v434: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 MiB/s wr, 6.12k op/s 2024-09-21T05:50:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.563+0000 7f94dd745640 1 -- 172.21.15.28:0/875862185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 msgr2=0x7f94d0094bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.563+0000 7f94dd745640 1 --2- 172.21.15.28:0/875862185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d0094bb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009900 tx=0x7f94cc02b640 comp rx=0 tx=0).stop 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.563+0000 7f94dd745640 1 -- 172.21.15.28:0/875862185 shutdown_connections 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 --2- 172.21.15.28:0/875862185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 0x7f94d00a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 --2- 172.21.15.28:0/875862185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d009a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 --2- 172.21.15.28:0/875862185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d0094bb0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 -- 172.21.15.28:0/875862185 >> 172.21.15.28:0/875862185 conn(0x7f94d000c4a0 msgr2=0x7f94d000c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 -- 172.21.15.28:0/875862185 shutdown_connections 2024-09-21T05:50:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.564+0000 7f94dd745640 1 -- 172.21.15.28:0/875862185 wait complete. 2024-09-21T05:50:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.566+0000 7f94dd745640 1 Processor -- start 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.566+0000 7f94dd745640 1 -- start start 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d012a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 0x7f94d012fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a3ec0 con 0x7f94d00947a0 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49678/0 (socket says 172.21.15.28:49678) 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94d77fe640 1 -- 172.21.15.28:0/3736117918 learned_addr learned my addr 172.21.15.28:0/3736117918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a3d40 con 0x7f94d009aa30 2024-09-21T05:50:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dcf44640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 0x7f94d012fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00a4040 con 0x7f94d009a0e0 2024-09-21T05:50:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.567+0000 7f94d7fff640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d012a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 -- 172.21.15.28:0/3736117918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 msgr2=0x7f94d012fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 0x7f94d012fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 -- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 msgr2=0x7f94d012a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d012a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d01305b0 con 0x7f94d009a0e0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d77fe640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f94c4010c00 tx=0x7f94c400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.568+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c401d070 con 0x7f94d009a0e0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.569+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d01308a0 con 0x7f94d009a0e0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.569+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94c4018ae0 con 0x7f94d009a0e0 2024-09-21T05:50:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.569+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c4016a90 con 0x7f94d009a0e0 2024-09-21T05:50:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.569+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d0137000 con 0x7f94d009a0e0 2024-09-21T05:50:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.570+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d012b770 con 0x7f94d009a0e0 2024-09-21T05:50:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.573+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f94c4020070 con 0x7f94d009a0e0 2024-09-21T05:50:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.574+0000 7f94d57fa640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 0x7f94c80b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.574+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f94c40e7ac0 con 0x7f94d009a0e0 2024-09-21T05:50:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.575+0000 7f94d7fff640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 0x7f94c80b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.577+0000 7f94d7fff640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 0x7f94c80b8c40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005f70 tx=0x7f94cc000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:40.580+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f94c40b2410 con 0x7f94d009a0e0 2024-09-21T05:50:41.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.030+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94d013b940 con 0x7f94d009a0e0 2024-09-21T05:50:41.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.031+0000 7f94d57fa640 1 -- 172.21.15.28:0/3736117918 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11997 (secure 0 0 0) 0x7f94c40b7350 con 0x7f94d009a0e0 2024-09-21T05:50:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:50:41.034 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"btime":"2024-09-21T05:50:29:562813+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:50:29.270500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:50:41.034 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-21T05:50:41.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 msgr2=0x7f94c80b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:41.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 0x7f94c80b8c40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005f70 tx=0x7f94cc000bc0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 msgr2=0x7f94d012abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:41.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f94c4010c00 tx=0x7f94c400c6a0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 shutdown_connections 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d009aa30 0x7f94d012fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.035+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94d009a0e0 0x7f94d012abe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.036+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f94c80b6730 0x7f94c80b8c40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.036+0000 7f94dd745640 1 --2- 172.21.15.28:0/3736117918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d00947a0 0x7f94d012a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.036+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 >> 172.21.15.28:0/3736117918 conn(0x7f94d000c4a0 msgr2=0x7f94d0097f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.036+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 shutdown_connections 2024-09-21T05:50:41.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:41.036+0000 7f94dd745640 1 -- 172.21.15.28:0/3736117918 wait complete. 2024-09-21T05:50:41.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:50:42.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:41 smithi181 ceph-mon[34819]: pgmap v435: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 5.40k op/s 2024-09-21T05:50:42.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:41 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3736117918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:50:42.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:41 smithi028 ceph-mon[31227]: pgmap v435: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 5.40k op/s 2024-09-21T05:50:42.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:41 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3736117918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:50:42.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:41 smithi129 ceph-mon[34885]: pgmap v435: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 5.40k op/s 2024-09-21T05:50:42.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:41 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3736117918' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:50:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.004+0000 7f5881542640 1 -- 172.21.15.28:0/1947557112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c108cf0 msgr2=0x7f587c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.004+0000 7f5881542640 1 --2- 172.21.15.28:0/1947557112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c108cf0 0x7f587c10f520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5874009ca0 tx=0x7f587402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 -- 172.21.15.28:0/1947557112 shutdown_connections 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 --2- 172.21.15.28:0/1947557112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c108cf0 0x7f587c10f520 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 --2- 172.21.15.28:0/1947557112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c1083a0 0x7f587c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 --2- 172.21.15.28:0/1947557112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c102a60 0x7f587c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 -- 172.21.15.28:0/1947557112 >> 172.21.15.28:0/1947557112 conn(0x7f587c079730 msgr2=0x7f587c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 -- 172.21.15.28:0/1947557112 shutdown_connections 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.005+0000 7f5881542640 1 -- 172.21.15.28:0/1947557112 wait complete. 2024-09-21T05:50:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.007+0000 7f5881542640 1 Processor -- start 2024-09-21T05:50:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.007+0000 7f5881542640 1 -- start start 2024-09-21T05:50:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 0x7f587c19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c1083a0 0x7f587c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f587c111860 con 0x7f587c108cf0 2024-09-21T05:50:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f587c1116e0 con 0x7f587c102a60 2024-09-21T05:50:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f5881542640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f587c1119e0 con 0x7f587c1083a0 2024-09-21T05:50:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f587affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 0x7f587c19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58082/0 (socket says 172.21.15.28:58082) 2024-09-21T05:50:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f587b7fe640 1 -- 172.21.15.28:0/3376609839 learned_addr learned my addr 172.21.15.28:0/3376609839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.008+0000 7f587a7fc640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c1083a0 0x7f587c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 -- 172.21.15.28:0/3376609839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c1083a0 msgr2=0x7f587c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c1083a0 0x7f587c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 -- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 msgr2=0x7f587c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 -- 172.21.15.28:0/3376609839 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587c1acb50 con 0x7f587c102a60 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587affd640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 0x7f587c19bd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f586c00c970 tx=0x7f586c00cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f5863fff640 1 -- 172.21.15.28:0/3376609839 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c007bb0 con 0x7f587c102a60 2024-09-21T05:50:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f587b7fe640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587c1acd20 con 0x7f587c102a60 2024-09-21T05:50:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.009+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f587c1ad220 con 0x7f587c102a60 2024-09-21T05:50:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.010+0000 7f5863fff640 1 -- 172.21.15.28:0/3376609839 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f586c007d50 con 0x7f587c102a60 2024-09-21T05:50:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.010+0000 7f5863fff640 1 -- 172.21.15.28:0/3376609839 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c018660 con 0x7f587c102a60 2024-09-21T05:50:43.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.014+0000 7f5887ab6740 1 Processor -- start 2024-09-21T05:50:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.014+0000 7f5887ab6740 1 -- start start 2024-09-21T05:50:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 0x564201819800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 0x564201819d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56420178fea0 con 0x564201472070 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56420177f2d0 con 0x564201451390 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5887ab6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5642017830e0 con 0x5642014491b0 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f5863fff640 1 -- 172.21.15.28:0/3376609839 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f586c018800 con 0x7f587c102a60 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58090/0 (socket says 172.21.15.28:58090) 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 0x564201819d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f587b7fe640 1 -- 172.21.15.28:0/851455913 learned_addr learned my addr 172.21.15.28:0/851455913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.015+0000 7f587affd640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 0x564201819800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587b7fe640 1 -- 172.21.15.28:0/851455913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 msgr2=0x564201819800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587b7fe640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 0x564201819800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587b7fe640 1 -- 172.21.15.28:0/851455913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 msgr2=0x564201819d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587b7fe640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 0x564201819d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587b7fe640 1 -- 172.21.15.28:0/851455913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642014dd650 con 0x564201472070 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587affd640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 0x564201819800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.016+0000 7f587a7fc640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 0x564201819d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.017+0000 7f587b7fe640 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f587402f390 tx=0x7f587402f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee25f52643a8c85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.017+0000 7f5863fff640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 0x7f58640b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.018+0000 7f5863fff640 1 -- 172.21.15.28:0/3376609839 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f586c0e7370 con 0x7f587c102a60 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.018+0000 7f587affd640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 0x7f58640b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.019+0000 7f58617fa640 1 -- 172.21.15.28:0/851455913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874031750 con 0x564201472070 2024-09-21T05:50:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.019+0000 7f58617fa640 1 -- 172.21.15.28:0/851455913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5874036040 con 0x564201472070 2024-09-21T05:50:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.019+0000 7f58617fa640 1 -- 172.21.15.28:0/851455913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874004710 con 0x564201472070 2024-09-21T05:50:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 -- 172.21.15.28:0/851455913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 msgr2=0x5642018192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f587402f390 tx=0x7f587402f3c0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f587affd640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 0x7f58640b8c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x564201817f40 tx=0x7f586c007f00 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 -- 172.21.15.28:0/851455913 shutdown_connections 2024-09-21T05:50:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201451390 0x564201819d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642014491b0 0x564201819800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 --2- 172.21.15.28:0/851455913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x5642018192c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 -- 172.21.15.28:0/851455913 >> 172.21.15.28:0/851455913 conn(0x564201466460 msgr2=0x5642018171b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 -- 172.21.15.28:0/851455913 shutdown_connections 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.020+0000 7f5887ab6740 1 -- 172.21.15.28:0/851455913 wait complete. 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.021+0000 7f5887ab6740 1 -- start start 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.021+0000 7f5887ab6740 1 Processor -- start 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x564201824e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58094/0 (socket says 172.21.15.28:58094) 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x564201824e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x564201824e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49702/0 (socket says 172.21.15.28:49702) 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201472070 0x564201825370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f587b7fe640 1 -- 172.21.15.28:0/2026363959 learned_addr learned my addr 172.21.15.28:0/2026363959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642015e0700 con 0x5642014491b0 2024-09-21T05:50:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56420171a160 con 0x564201472070 2024-09-21T05:50:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.022+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5642018278c0 con 0x564201451390 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.023+0000 7f5887ab6740 10 client.0 _ll_register_callbacks cb 0x5642017339e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.023+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201472070 0x564201825370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 -- 172.21.15.28:0/2026363959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 msgr2=0x564201824e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x564201824e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 -- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201472070 msgr2=0x564201825370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201472070 0x564201825370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564201747c90 con 0x5642014491b0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5874009c70 tx=0x7f58740043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeeaf429cadfd959 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.024+0000 7f585eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874003db0 con 0x5642014491b0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5874036040 con 0x5642014491b0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5874003f50 con 0x5642014491b0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f587402f660 con 0x5642014491b0 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f5887ab6740 10 client.16500 fetch_fsmap learned FSMap version 35 2024-09-21T05:50:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.025+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5642018444a0 con 0x5642014491b0 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f587402fcb0 con 0x5642014491b0 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8390+0+0 (secure 0 0 0) 0x7f5874044070 con 0x5642014491b0 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 10 client.16500 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 10 client.16500 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 20 client.16500 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5642014309f0 0x5642018479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 4 client.16500 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5874036040 con 0x5642014309f0 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.026+0000 7f587a7fc640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5642014309f0 0x5642018479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.027+0000 7f585eff5640 10 client.16500 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:50:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.027+0000 7f587a7fc640 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5642014309f0 0x5642018479a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2c4c28e946f3bf16 server_cookie=f3bc3f4248cd1cdf in_seq=0 out_seq=0 2024-09-21T05:50:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.028+0000 7f585eff5640 1 -- 172.21.15.28:0/2026363959 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f5874036040 con 0x5642014309f0 2024-09-21T05:50:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.028+0000 7f585eff5640 10 client.16500 handle_command_reply: tid=0 2024-09-21T05:50:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.028+0000 7f5887ab6740 1 client.16500 shutdown 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.028+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 msgr2=0x5642018248f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.028+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5874009c70 tx=0x7f58740043b0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 shutdown_connections 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5642014309f0 0x5642018479a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564201472070 0x564201825370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x564201824e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2026363959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642014491b0 0x5642018248f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 >> 172.21.15.28:0/2026363959 conn(0x564201466460 msgr2=0x56420181c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f585eff5640 0 client.16500 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:50:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 shutdown_connections 2024-09-21T05:50:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 1 -- 172.21.15.28:0/2026363959 wait complete. 2024-09-21T05:50:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.029+0000 7f5887ab6740 20 client.16500 trim_cache size 0 max 16384 2024-09-21T05:50:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 Processor -- start 2024-09-21T05:50:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 -- start start 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x56420184d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x56420184d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f586800e830 con 0x564201472070 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5868010490 con 0x5642014491b0 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f5887ab6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f586800ef00 con 0x564201451390 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.055+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:32814/0 (socket says 172.21.15.28:32814) 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 -- 172.21.15.28:0/1269090190 learned_addr learned my addr 172.21.15.28:0/1269090190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x56420184d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587a7fc640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x56420184d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 -- 172.21.15.28:0/1269090190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 msgr2=0x56420184d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x56420184d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 -- 172.21.15.28:0/1269090190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 msgr2=0x56420184d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x56420184d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587b7fe640 1 -- 172.21.15.28:0/1269090190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564201430c20 con 0x5642014491b0 2024-09-21T05:50:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.056+0000 7f587a7fc640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x56420184d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:50:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.057+0000 7f587b7fe640 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5874009620 tx=0x7f5874000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ab4e7402aa1ebba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.057+0000 7f5860ff9640 1 -- 172.21.15.28:0/1269090190 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874064070 con 0x5642014491b0 2024-09-21T05:50:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.057+0000 7f5860ff9640 1 -- 172.21.15.28:0/1269090190 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5874065040 con 0x5642014491b0 2024-09-21T05:50:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.057+0000 7f5860ff9640 1 -- 172.21.15.28:0/1269090190 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58740040a0 con 0x5642014491b0 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.058+0000 7f5887ab6740 1 -- 172.21.15.28:0/1269090190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 msgr2=0x564201826520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.058+0000 7f5887ab6740 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5874009620 tx=0x7f5874000f40 comp rx=0 tx=0).stop 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 -- 172.21.15.28:0/1269090190 shutdown_connections 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201472070 0x56420184d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201451390 0x56420184d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 --2- 172.21.15.28:0/1269090190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x564201826520 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 -- 172.21.15.28:0/1269090190 >> 172.21.15.28:0/1269090190 conn(0x564201466460 msgr2=0x564201825e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 -- 172.21.15.28:0/1269090190 shutdown_connections 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.059+0000 7f5887ab6740 1 -- 172.21.15.28:0/1269090190 wait complete. 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 -- start start 2024-09-21T05:50:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 Processor -- start 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201451390 0x5642018397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201472070 0x564201839cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f586800fd00 con 0x564201451390 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f586800f630 con 0x5642014491b0 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f586800c810 con 0x564201472070 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f5887ab6740 10 client.0 _ll_register_callbacks cb 0x7f58740048e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f587b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:32826/0 (socket says 172.21.15.28:32826) 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.060+0000 7f587b7fe640 1 -- 172.21.15.28:0/2906517579 learned_addr learned my addr 172.21.15.28:0/2906517579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201472070 0x564201839cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587a7fc640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201451390 0x5642018397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 -- 172.21.15.28:0/2906517579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201472070 msgr2=0x564201839cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201472070 0x564201839cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 -- 172.21.15.28:0/2906517579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201451390 msgr2=0x5642018397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201451390 0x5642018397b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.061+0000 7f587b7fe640 1 -- 172.21.15.28:0/2906517579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56420183a360 con 0x5642014491b0 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.062+0000 7f587b7fe640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f587404c040 tx=0x7f587402a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a622801fad8bdf35 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.062+0000 7f585eff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.063+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874064070 con 0x5642014491b0 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.063+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f5874003fb0 con 0x5642014491b0 2024-09-21T05:50:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.063+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5874065040 con 0x5642014491b0 2024-09-21T05:50:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.064+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5874004e10 con 0x5642014491b0 2024-09-21T05:50:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.064+0000 7f5887ab6740 10 client.26078 fetch_fsmap learned FSMap version 35 2024-09-21T05:50:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.064+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5642018c3150 con 0x5642014491b0 2024-09-21T05:50:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.065+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5874065040 con 0x5642014491b0 2024-09-21T05:50:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 35) ==== 8390+0+0 (secure 0 0 0) 0x7f5874044040 con 0x5642014491b0 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 10 client.26078 fetch_fsmap finished waiting for FSMap version 35 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 10 client.26078 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 20 client.26078 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x564201405da0 0x5642018c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 4 client.26078 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564201791df0 con 0x564201405da0 2024-09-21T05:50:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.066+0000 7f587a7fc640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x564201405da0 0x5642018c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.068+0000 7f587a7fc640 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x564201405da0 0x5642018c57d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=58ca95bad52f5abb server_cookie=89b8f3bf7433ccab in_seq=0 out_seq=0 2024-09-21T05:50:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.068+0000 7f585eff5640 10 client.26078 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:50:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.068+0000 7f585eff5640 1 -- 172.21.15.28:0/2906517579 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564201791df0 con 0x564201405da0 2024-09-21T05:50:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.068+0000 7f585eff5640 10 client.26078 handle_command_reply: tid=0 2024-09-21T05:50:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.068+0000 7f5887ab6740 1 client.26078 shutdown 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 msgr2=0x56420183e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f587404c040 tx=0x7f587402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 shutdown_connections 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x564201405da0 0x5642018c57d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f585eff5640 0 client.26078 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564201472070 0x564201839cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564201451390 0x5642018397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 --2- 172.21.15.28:0/2906517579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642014491b0 0x56420183e300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 >> 172.21.15.28:0/2906517579 conn(0x564201466460 msgr2=0x56420181c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 shutdown_connections 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 1 -- 172.21.15.28:0/2906517579 wait complete. 2024-09-21T05:50:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.069+0000 7f5887ab6740 20 client.26078 trim_cache size 0 max 16384 2024-09-21T05:50:43.073 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:50:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 msgr2=0x7f58640b8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 0x7f58640b8c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x564201817f40 tx=0x7f586c007f00 comp rx=0 tx=0).stop 2024-09-21T05:50:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 msgr2=0x7f587c19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 0x7f587c19bd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f586c00c970 tx=0x7f586c00cd70 comp rx=0 tx=0).stop 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 shutdown_connections 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f58640b6770 0x7f58640b8c80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f587c108cf0 0x7f587c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f587c1083a0 0x7f587c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 --2- 172.21.15.28:0/3376609839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f587c102a60 0x7f587c19bd20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 >> 172.21.15.28:0/3376609839 conn(0x7f587c079730 msgr2=0x7f587c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 shutdown_connections 2024-09-21T05:50:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:43.071+0000 7f5881542640 1 -- 172.21.15.28:0/3376609839 wait complete. 2024-09-21T05:50:44.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:43 smithi028 ceph-mon[31227]: pgmap v436: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 55 MiB/s wr, 7.41k op/s 2024-09-21T05:50:44.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:50:44.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:43 smithi181 ceph-mon[34819]: pgmap v436: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 55 MiB/s wr, 7.41k op/s 2024-09-21T05:50:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:43 smithi129 ceph-mon[34885]: pgmap v436: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 55 MiB/s wr, 7.41k op/s 2024-09-21T05:50:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:45.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.376+0000 7fcced832640 1 -- 172.21.15.28:0/3081101150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce81049b0 msgr2=0x7fcce8104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.376+0000 7fcced832640 1 --2- 172.21.15.28:0/3081101150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce81049b0 0x7fcce8104dc0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009ca0 tx=0x7fcce002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.377+0000 7fcced832640 1 -- 172.21.15.28:0/3081101150 shutdown_connections 2024-09-21T05:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.377+0000 7fcced832640 1 --2- 172.21.15.28:0/3081101150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce810de10 0x7fcce8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.377+0000 7fcced832640 1 --2- 172.21.15.28:0/3081101150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 0x7fcce810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.377+0000 7fcced832640 1 --2- 172.21.15.28:0/3081101150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce81049b0 0x7fcce8104dc0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.377+0000 7fcced832640 1 -- 172.21.15.28:0/3081101150 >> 172.21.15.28:0/3081101150 conn(0x7fcce8079730 msgr2=0x7fcce8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.378+0000 7fcced832640 1 -- 172.21.15.28:0/3081101150 shutdown_connections 2024-09-21T05:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.378+0000 7fcced832640 1 -- 172.21.15.28:0/3081101150 wait complete. 2024-09-21T05:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.379+0000 7fcced832640 1 Processor -- start 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 -- start start 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce81049b0 0x7fcce810ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 0x7fcce810b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8115ca0 con 0x7fcce810de10 2024-09-21T05:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8115b20 con 0x7fcce81049b0 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcced832640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8115e20 con 0x7fcce8105300 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43374/0 (socket says 172.21.15.28:43374) 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.380+0000 7fcce77fe640 1 -- 172.21.15.28:0/3971940356 learned_addr learned my addr 172.21.15.28:0/3971940356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 0x7fcce810b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce81049b0 0x7fcce810ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce77fe640 1 -- 172.21.15.28:0/3971940356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 msgr2=0x7fcce810b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce77fe640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 0x7fcce810b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce77fe640 1 -- 172.21.15.28:0/3971940356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce81049b0 msgr2=0x7fcce810ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce77fe640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce81049b0 0x7fcce810ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.381+0000 7fcce77fe640 1 -- 172.21.15.28:0/3971940356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce81a2480 con 0x7fcce810de10 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fcce77fe640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e9b0 tx=0x7fccdc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fccec830640 1 -- 172.21.15.28:0/3971940356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00cd80 con 0x7fcce810de10 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fccec830640 1 -- 172.21.15.28:0/3971940356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccdc0035f0 con 0x7fcce810de10 2024-09-21T05:50:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce81a2770 con 0x7fcce810de10 2024-09-21T05:50:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce81a2cb0 con 0x7fcce810de10 2024-09-21T05:50:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.382+0000 7fccec830640 1 -- 172.21.15.28:0/3971940356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc010690 con 0x7fcce810de10 2024-09-21T05:50:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.387+0000 7fccec830640 1 -- 172.21.15.28:0/3971940356 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fccdc010830 con 0x7fcce810de10 2024-09-21T05:50:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.389+0000 7fccec830640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 0x7fccd00b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.389+0000 7fccf3da6740 1 Processor -- start 2024-09-21T05:50:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.389+0000 7fccf3da6740 1 -- start start 2024-09-21T05:50:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.389+0000 7fccec830640 1 -- 172.21.15.28:0/3971940356 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fccdc0eb060 con 0x7fcce810de10 2024-09-21T05:50:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.389+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 0x7fccd00b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07176a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0717be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f0718120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590f068dea0 con 0x5590f03471b0 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5590f067d2d0 con 0x5590f034f390 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fccf3da6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5590f06810e0 con 0x5590f0370070 2024-09-21T05:50:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f0718120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 0x7fccd00b8ce0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcce0000f80 tx=0x7fcce0009800 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fcce6ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07176a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f0718120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36918/0 (socket says 172.21.15.28:36918) 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.390+0000 7fcce77fe640 1 -- 172.21.15.28:0/2719777121 learned_addr learned my addr 172.21.15.28:0/2719777121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce67fc640 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0717be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce6ffd640 1 -- 172.21.15.28:0/2719777121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 msgr2=0x5590f0718120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f0718120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce6ffd640 1 -- 172.21.15.28:0/2719777121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 msgr2=0x5590f0717be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0717be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.391+0000 7fcce6ffd640 1 -- 172.21.15.28:0/2719777121 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590f03db650 con 0x5590f0370070 2024-09-21T05:50:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.392+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07176a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcce810d8b0 tx=0x7fcce003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c76f03b97518aca server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.392+0000 7fcccdffb640 1 -- 172.21.15.28:0/2719777121 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0003970 con 0x5590f0370070 2024-09-21T05:50:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.392+0000 7fcccdffb640 1 -- 172.21.15.28:0/2719777121 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce0036040 con 0x5590f0370070 2024-09-21T05:50:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.392+0000 7fcccdffb640 1 -- 172.21.15.28:0/2719777121 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce002f8c0 con 0x5590f0370070 2024-09-21T05:50:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 -- 172.21.15.28:0/2719777121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 msgr2=0x5590f07176a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07176a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcce810d8b0 tx=0x7fcce003e000 comp rx=0 tx=0).stop 2024-09-21T05:50:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 -- 172.21.15.28:0/2719777121 shutdown_connections 2024-09-21T05:50:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f0718120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0717be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 --2- 172.21.15.28:0/2719777121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07176a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.394+0000 7fccf3da6740 1 -- 172.21.15.28:0/2719777121 >> 172.21.15.28:0/2719777121 conn(0x5590f0364460 msgr2=0x5590f0715590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.396+0000 7fccf3da6740 1 -- 172.21.15.28:0/2719777121 shutdown_connections 2024-09-21T05:50:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.397+0000 7fccf3da6740 1 -- 172.21.15.28:0/2719777121 wait complete. 2024-09-21T05:50:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.398+0000 7fccf3da6740 1 -- start start 2024-09-21T05:50:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.398+0000 7fccf3da6740 1 Processor -- start 2024-09-21T05:50:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 0x5590f0722d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07232c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 0x5590f072f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fccf3da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590f04de700 con 0x5590f034f390 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fcce67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 0x5590f072f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fcce77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 0x5590f072f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36926/0 (socket says 172.21.15.28:36926) 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fcce77fe640 1 -- 172.21.15.28:0/3746545086 learned_addr learned my addr 172.21.15.28:0/3746545086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 0x5590f0722d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.399+0000 7fcce67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07232c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43384/0 (socket says 172.21.15.28:43384) 2024-09-21T05:50:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5590f0618160 con 0x5590f0370070 2024-09-21T05:50:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5590f0725e20 con 0x5590f03471b0 2024-09-21T05:50:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.400+0000 7fccf3da6740 10 client.0 _ll_register_callbacks cb 0x5590f06319e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.401+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3746545086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 msgr2=0x5590f072f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.401+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 0x5590f072f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.401+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3746545086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 msgr2=0x5590f07232c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.401+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07232c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.401+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590f0645c90 con 0x5590f03471b0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 0x5590f0722d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcce002ade0 tx=0x7fcce0005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6afe8e8b30c0b78 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0060060 con 0x5590f03471b0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcce0004040 con 0x5590f03471b0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce0036040 con 0x5590f03471b0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcce0004240 con 0x5590f03471b0 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccf3da6740 10 client.26113 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.402+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590f0731b80 con 0x5590f03471b0 2024-09-21T05:50:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce00043e0 con 0x5590f03471b0 2024-09-21T05:50:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fcce00305c0 con 0x5590f03471b0 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 10 client.26113 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 10 client.26113 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 20 client.26113 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f032e9f0 0x5590f0734ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 4 client.26113 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcce0036040 con 0x5590f032e9f0 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.403+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f032e9f0 0x5590f0734ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.404+0000 7fccc6ffd640 10 client.26113 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.404+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f032e9f0 0x5590f0734ee0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73ff6a413b35faaa server_cookie=60ce49f38b7aa66e in_seq=0 out_seq=0 2024-09-21T05:50:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.406+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3746545086 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fcce0036040 con 0x5590f032e9f0 2024-09-21T05:50:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.406+0000 7fccc6ffd640 10 client.26113 handle_command_reply: tid=0 2024-09-21T05:50:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.406+0000 7fccf3da6740 1 client.26113 shutdown 2024-09-21T05:50:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.406+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 msgr2=0x5590f0722d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.406+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 0x5590f0722d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcce002ade0 tx=0x7fcce0005d80 comp rx=0 tx=0).stop 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 shutdown_connections 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f032e9f0 0x5590f0734ee0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f0370070 0x5590f072f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07232c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3746545086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f03471b0 0x5590f0722d80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccc6ffd640 0 client.26113 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:50:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.407+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 >> 172.21.15.28:0/3746545086 conn(0x5590f0364460 msgr2=0x5590f071a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.408+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 shutdown_connections 2024-09-21T05:50:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.408+0000 7fccf3da6740 1 -- 172.21.15.28:0/3746545086 wait complete. 2024-09-21T05:50:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.408+0000 7fccf3da6740 20 client.26113 trim_cache size 0 max 16384 2024-09-21T05:50:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.433+0000 7fccf3da6740 1 Processor -- start 2024-09-21T05:50:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.433+0000 7fccf3da6740 1 -- start start 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f072f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fcce6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fcce6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43398/0 (socket says 172.21.15.28:43398) 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3823392545 learned_addr learned my addr 172.21.15.28:0/3823392545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f0714bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f072f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400e860 con 0x5590f03471b0 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fccd40104f0 con 0x5590f034f390 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.434+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400ef30 con 0x5590f0370070 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce77fe640 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f0714bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3823392545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 msgr2=0x5590f0714bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f0714bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3823392545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 msgr2=0x5590f072f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f072f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3823392545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590f0721280 con 0x5590f03471b0 2024-09-21T05:50:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.435+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcce0031ba0 tx=0x7fcce0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3a73582fee726d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.436+0000 7fcccd7fa640 1 -- 172.21.15.28:0/3823392545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0060070 con 0x5590f03471b0 2024-09-21T05:50:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.436+0000 7fcccd7fa640 1 -- 172.21.15.28:0/3823392545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce0036040 con 0x5590f03471b0 2024-09-21T05:50:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.436+0000 7fcccd7fa640 1 -- 172.21.15.28:0/3823392545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce00043e0 con 0x5590f03471b0 2024-09-21T05:50:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 msgr2=0x5590f0723240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcce0031ba0 tx=0x7fcce0005df0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 shutdown_connections 2024-09-21T05:50:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f0714bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f034f390 0x5590f072f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3823392545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f03471b0 0x5590f0723240 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.437+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 >> 172.21.15.28:0/3823392545 conn(0x5590f0364460 msgr2=0x5590f0722b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.438+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 shutdown_connections 2024-09-21T05:50:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.438+0000 7fccf3da6740 1 -- 172.21.15.28:0/3823392545 wait complete. 2024-09-21T05:50:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.438+0000 7fccf3da6740 1 -- start start 2024-09-21T05:50:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 Processor -- start 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07364b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400fd60 con 0x5590f034f390 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400f660 con 0x5590f03471b0 2024-09-21T05:50:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.439+0000 7fccf3da6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400c840 con 0x5590f0370070 2024-09-21T05:50:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fccf3da6740 10 client.0 _ll_register_callbacks cb 0x7fccdc0e8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36944/0 (socket says 172.21.15.28:36944) 2024-09-21T05:50:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3909261599 learned_addr learned my addr 172.21.15.28:0/3909261599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce77fe640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 msgr2=0x5590f07369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3909261599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 msgr2=0x5590f07364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.440+0000 7fcce6ffd640 1 -- 172.21.15.28:0/3909261599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590f079e6e0 con 0x5590f03471b0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.441+0000 7fcce6ffd640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcce002ade0 tx=0x7fcce0009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7f0ec115474aff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.441+0000 7fccc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.442+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0072020 con 0x5590f03471b0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.442+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fcce00043e0 con 0x5590f03471b0 2024-09-21T05:50:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.442+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce0036040 con 0x5590f03471b0 2024-09-21T05:50:45.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.443+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcce0060040 con 0x5590f03471b0 2024-09-21T05:50:45.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.443+0000 7fccf3da6740 10 client.26096 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:45.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.443+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590f079f940 con 0x5590f03471b0 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce0036040 con 0x5590f03471b0 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fcce006d050 con 0x5590f03471b0 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 10 client.26096 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 10 client.26096 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 20 client.26096 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f0303da0 0x5590f07a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 4 client.26096 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:50:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.444+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5590f068fdf0 con 0x5590f0303da0 2024-09-21T05:50:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.445+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f0303da0 0x5590f07a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.445+0000 7fccc6ffd640 10 client.26096 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:50:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.445+0000 7fcce67fc640 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f0303da0 0x5590f07a1ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d781e643c46f8e7 server_cookie=908d9aa13f11b701 in_seq=0 out_seq=0 2024-09-21T05:50:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.446+0000 7fccc6ffd640 1 -- 172.21.15.28:0/3909261599 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5590f068fdf0 con 0x5590f0303da0 2024-09-21T05:50:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.446+0000 7fccc6ffd640 10 client.26096 handle_command_reply: tid=0 2024-09-21T05:50:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.446+0000 7fccf3da6740 1 client.26096 shutdown 2024-09-21T05:50:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.446+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 msgr2=0x5590f0735f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.446+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcce002ade0 tx=0x7fcce0009360 comp rx=0 tx=0).stop 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 shutdown_connections 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x5590f0303da0 0x5590f07a1ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5590f0370070 0x5590f07369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590f034f390 0x5590f07364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 --2- 172.21.15.28:0/3909261599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5590f03471b0 0x5590f0735f70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 >> 172.21.15.28:0/3909261599 conn(0x5590f0364460 msgr2=0x5590f07241a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccc6ffd640 0 client.26096 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 shutdown_connections 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.447+0000 7fccf3da6740 1 -- 172.21.15.28:0/3909261599 wait complete. 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.448+0000 7fccf3da6740 20 client.26096 trim_cache size 0 max 16384 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:50:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.448+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 msgr2=0x7fccd00b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 0x7fccd00b8ce0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcce0000f80 tx=0x7fcce0009800 comp rx=0 tx=0).stop 2024-09-21T05:50:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 msgr2=0x7fcce810b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e9b0 tx=0x7fccdc00ee70 comp rx=0 tx=0).stop 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 shutdown_connections 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fccd00b67d0 0x7fccd00b8ce0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce810de10 0x7fcce810b8f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcce8105300 0x7fcce810b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 --2- 172.21.15.28:0/3971940356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcce81049b0 0x7fcce810ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 >> 172.21.15.28:0/3971940356 conn(0x7fcce8079730 msgr2=0x7fcce8100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 shutdown_connections 2024-09-21T05:50:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:45.449+0000 7fcced832640 1 -- 172.21.15.28:0/3971940356 wait complete. 2024-09-21T05:50:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.978+0000 7f07c628c640 1 -- 172.21.15.28:0/4286512016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 msgr2=0x7f07c0102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.978+0000 7f07c628c640 1 --2- 172.21.15.28:0/4286512016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 0x7f07c0102470 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009860 tx=0x7f07b002a380 comp rx=0 tx=0).stop 2024-09-21T05:50:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.984+0000 7f07c628c640 1 -- 172.21.15.28:0/4286512016 shutdown_connections 2024-09-21T05:50:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.984+0000 7f07c628c640 1 --2- 172.21.15.28:0/4286512016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107b70 0x7f07c0108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.984+0000 7f07c628c640 1 --2- 172.21.15.28:0/4286512016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107220 0x7f07c0107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.984+0000 7f07c628c640 1 --2- 172.21.15.28:0/4286512016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 0x7f07c0102470 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.984+0000 7f07c628c640 1 -- 172.21.15.28:0/4286512016 >> 172.21.15.28:0/4286512016 conn(0x7f07c0079e90 msgr2=0x7f07c007a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.988+0000 7f07c628c640 1 -- 172.21.15.28:0/4286512016 shutdown_connections 2024-09-21T05:50:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.989+0000 7f07c628c640 1 -- 172.21.15.28:0/4286512016 wait complete. 2024-09-21T05:50:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.990+0000 7f07c628c640 1 Processor -- start 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.991+0000 7f07c628c640 1 -- start start 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 0x7f07c0074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107220 0x7f07c006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07c011a0c0 con 0x7f07c0102060 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0119f40 con 0x7f07c0107220 2024-09-21T05:50:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c628c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07c011a240 con 0x7f07c0107b70 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c4829640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c4829640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46594/0 (socket says 172.21.15.28:46594) 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07c4829640 1 -- 172.21.15.28:0/4094992652 learned_addr learned my addr 172.21.15.28:0/4094992652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.992+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107220 0x7f07c006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 -- 172.21.15.28:0/4094992652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107220 msgr2=0x7f07c006e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107220 0x7f07c006e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 -- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 msgr2=0x7f07c0074ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 0x7f07c0074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 -- 172.21.15.28:0/4094992652 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c006efa0 con 0x7f07c0107b70 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.993+0000 7f07c4829640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fd0 tx=0x7f07b800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.994+0000 7f07bd7fa640 1 -- 172.21.15.28:0/4094992652 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004b20 con 0x7f07c0107b70 2024-09-21T05:50:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.994+0000 7f07bd7fa640 1 -- 172.21.15.28:0/4094992652 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b8003e20 con 0x7f07c0107b70 2024-09-21T05:50:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.994+0000 7f07bd7fa640 1 -- 172.21.15.28:0/4094992652 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004820 con 0x7f07c0107b70 2024-09-21T05:50:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.994+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c0073c50 con 0x7f07c0107b70 2024-09-21T05:50:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:45.994+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c0074120 con 0x7f07c0107b70 2024-09-21T05:50:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.000+0000 7f07cd1f4740 1 Processor -- start 2024-09-21T05:50:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.004+0000 7f07cd1f4740 1 -- start start 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.004+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355a92f0 0x564335977ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355b14d0 0x564335978430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643358ec6f0 con 0x5643355a92f0 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5643358dbb20 con 0x5643355d21b0 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5643358df930 con 0x5643355b14d0 2024-09-21T05:50:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.005+0000 7f07bd7fa640 1 -- 172.21.15.28:0/4094992652 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f07b801d020 con 0x7f07c0107b70 2024-09-21T05:50:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36970/0 (socket says 172.21.15.28:36970) 2024-09-21T05:50:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 -- 172.21.15.28:0/832105417 learned_addr learned my addr 172.21.15.28:0/832105417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 -- 172.21.15.28:0/832105417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355b14d0 msgr2=0x564335978430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355b14d0 0x564335978430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 -- 172.21.15.28:0/832105417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355a92f0 msgr2=0x564335977ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355a92f0 0x564335977ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.006+0000 7f07bf7fe640 1 -- 172.21.15.28:0/832105417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643358a1530 con 0x5643355d21b0 2024-09-21T05:50:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.010+0000 7f07bd7fa640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 0x7f07a80b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.011+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009a90 tx=0x7f07b4009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8d76674e7c8691 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.011+0000 7f07bffff640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 0x7f07a80b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.012+0000 7f07bffff640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 0x7f07a80b8cc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f07b00095f0 tx=0x7f07b002a880 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.012+0000 7f07bd7fa640 1 -- 172.21.15.28:0/4094992652 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f07b80e7620 con 0x7f07c0107b70 2024-09-21T05:50:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.015+0000 7f07a67fc640 1 -- 172.21.15.28:0/832105417 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4018020 con 0x5643355d21b0 2024-09-21T05:50:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.015+0000 7f07a67fc640 1 -- 172.21.15.28:0/832105417 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b4010dc0 con 0x5643355d21b0 2024-09-21T05:50:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.015+0000 7f07a67fc640 1 -- 172.21.15.28:0/832105417 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4013690 con 0x5643355d21b0 2024-09-21T05:50:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.017+0000 7f07cd1f4740 1 -- 172.21.15.28:0/832105417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 msgr2=0x5643359779b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.017+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009a90 tx=0x7f07b4009f50 comp rx=0 tx=0).stop 2024-09-21T05:50:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.021+0000 7f07cd1f4740 1 -- 172.21.15.28:0/832105417 shutdown_connections 2024-09-21T05:50:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.021+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355b14d0 0x564335978430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.021+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355a92f0 0x564335977ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.021+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/832105417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359779b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.021+0000 7f07cd1f4740 1 -- 172.21.15.28:0/832105417 >> 172.21.15.28:0/832105417 conn(0x5643355c6300 msgr2=0x5643359758a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.030+0000 7f07cd1f4740 1 -- 172.21.15.28:0/832105417 shutdown_connections 2024-09-21T05:50:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.032+0000 7f07cd1f4740 1 -- 172.21.15.28:0/832105417 wait complete. 2024-09-21T05:50:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.032+0000 7f07cd1f4740 1 -- start start 2024-09-21T05:50:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.033+0000 7f07cd1f4740 1 Processor -- start 2024-09-21T05:50:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335983630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564335813ab0 con 0x5643355b14d0 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564335881730 con 0x5643355d21b0 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564335986130 con 0x5643355a92f0 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.037+0000 7f07cd1f4740 10 client.0 _ll_register_callbacks cb 0x5643358d56b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.038+0000 7f07bffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.038+0000 7f07bffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46604/0 (socket says 172.21.15.28:46604) 2024-09-21T05:50:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.038+0000 7f07bffff640 1 -- 172.21.15.28:0/384153790 learned_addr learned my addr 172.21.15.28:0/384153790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.039+0000 7f07bffff640 1 -- 172.21.15.28:0/384153790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 msgr2=0x56433598f4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:50:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.039+0000 7f07bffff640 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.039+0000 7f07bffff640 1 -- 172.21.15.28:0/384153790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 msgr2=0x564335983630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.039+0000 7f07bffff640 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335983630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.039+0000 7f07bffff640 1 -- 172.21.15.28:0/384153790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56433563d790 con 0x5643355a92f0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.043+0000 7f07bffff640 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009250 tx=0x7f07b0030840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e270f18d404dade server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.043+0000 7f079f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.044+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0003800 con 0x5643355a92f0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.045+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07b4010dc0 con 0x5643355a92f0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.045+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f07b00039a0 con 0x5643355a92f0 2024-09-21T05:50:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.045+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f07b0003ba0 con 0x5643355a92f0 2024-09-21T05:50:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.049+0000 7f07cd1f4740 10 client.26125 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.049+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5643356b6620 con 0x5643355a92f0 2024-09-21T05:50:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.052+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b4010dc0 con 0x5643355a92f0 2024-09-21T05:50:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.052+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f07b003e430 con 0x5643355a92f0 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 10 client.26125 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 10 client.26125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 20 client.26125 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564335590d80 0x5643359954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 4 client.26125 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.058+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643358ee640 con 0x564335590d80 2024-09-21T05:50:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.063+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564335590d80 0x5643359954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.072+0000 7f07bf7fe640 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564335590d80 0x5643359954b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f87c232e59e6c7f6 server_cookie=9b89ac3c68f12a13 in_seq=0 out_seq=0 2024-09-21T05:50:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.073+0000 7f079f7fe640 10 client.26125 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.084+0000 7f079f7fe640 1 -- 172.21.15.28:0/384153790 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5643358ee640 con 0x564335590d80 2024-09-21T05:50:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.084+0000 7f079f7fe640 10 client.26125 handle_command_reply: tid=0 2024-09-21T05:50:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.085+0000 7f07cd1f4740 1 client.26125 shutdown 2024-09-21T05:50:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.088+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 msgr2=0x5643359830f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.088+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009250 tx=0x7f07b0030840 comp rx=0 tx=0).stop 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 shutdown_connections 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564335590d80 0x5643359954b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335983630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/384153790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359830f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 >> 172.21.15.28:0/384153790 conn(0x5643355c6300 msgr2=0x5643359789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.091+0000 7f079f7fe640 0 client.26125 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.095+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 shutdown_connections 2024-09-21T05:50:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.096+0000 7f07cd1f4740 1 -- 172.21.15.28:0/384153790 wait complete. 2024-09-21T05:50:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.099+0000 7f07cd1f4740 20 client.26125 trim_cache size 0 max 16384 2024-09-21T05:50:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 Processor -- start 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 -- start start 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359835d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x5643359790f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac00e8c0 con 0x5643355b14d0 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac010550 con 0x5643355d21b0 2024-09-21T05:50:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.133+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac00ef90 con 0x5643355a92f0 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.134+0000 7f07c4829640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.134+0000 7f07c4829640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36984/0 (socket says 172.21.15.28:36984) 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.134+0000 7f07c4829640 1 -- 172.21.15.28:0/3718928865 learned_addr learned my addr 172.21.15.28:0/3718928865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07bffff640 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359835d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07c4829640 1 -- 172.21.15.28:0/3718928865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 msgr2=0x5643359835d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07c4829640 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359835d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07c4829640 1 -- 172.21.15.28:0/3718928865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 msgr2=0x5643359790f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07c4829640 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x5643359790f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.135+0000 7f07c4829640 1 -- 172.21.15.28:0/3718928865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564335990290 con 0x5643355d21b0 2024-09-21T05:50:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.138+0000 7f07c4829640 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fa0 tx=0x7f07b80ace40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec8b4e02a93398fe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.141+0000 7f07a5ffb640 1 -- 172.21.15.28:0/3718928865 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b801a040 con 0x5643355d21b0 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.141+0000 7f07a5ffb640 1 -- 172.21.15.28:0/3718928865 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b80b1f10 con 0x5643355d21b0 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.141+0000 7f07a5ffb640 1 -- 172.21.15.28:0/3718928865 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b80a7b30 con 0x5643355d21b0 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 -- 172.21.15.28:0/3718928865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 msgr2=0x56433598f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fa0 tx=0x7f07b80ace40 comp rx=0 tx=0).stop 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 -- 172.21.15.28:0/3718928865 shutdown_connections 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x56433598f3b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x5643359790f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/3718928865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x5643359835d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.143+0000 7f07cd1f4740 1 -- 172.21.15.28:0/3718928865 >> 172.21.15.28:0/3718928865 conn(0x5643355c6300 msgr2=0x5643359789e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.149+0000 7f07cd1f4740 1 -- 172.21.15.28:0/3718928865 shutdown_connections 2024-09-21T05:50:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.149+0000 7f07cd1f4740 1 -- 172.21.15.28:0/3718928865 wait complete. 2024-09-21T05:50:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.149+0000 7f07cd1f4740 1 -- start start 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 Processor -- start 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x564335986ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335987480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac00fdc0 con 0x5643355b14d0 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac00f6c0 con 0x5643355d21b0 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac00c8a0 con 0x5643355a92f0 2024-09-21T05:50:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.150+0000 7f07cd1f4740 10 client.0 _ll_register_callbacks cb 0x7f07b0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.151+0000 7f07bffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x564335986ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.151+0000 7f07bffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x564335986ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46622/0 (socket says 172.21.15.28:46622) 2024-09-21T05:50:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07bffff640 1 -- 172.21.15.28:0/2081978897 learned_addr learned my addr 172.21.15.28:0/2081978897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 -- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 msgr2=0x564335986ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x564335986ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 -- 172.21.15.28:0/2081978897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 msgr2=0x564335987480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335987480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.152+0000 7f07c4829640 1 -- 172.21.15.28:0/2081978897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643359f8610 con 0x5643355d21b0 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.153+0000 7f07c4829640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359f7f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f07b80ad130 tx=0x7f07b80ad0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1542ea27e32a7f06 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.153+0000 7f079f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.153+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b801a040 con 0x5643355d21b0 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.153+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07b80b1f10 con 0x5643355d21b0 2024-09-21T05:50:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.153+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f07b8094d70 con 0x5643355d21b0 2024-09-21T05:50:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.156+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f07b80c8050 con 0x5643355d21b0 2024-09-21T05:50:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.156+0000 7f07cd1f4740 10 client.26111 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.156+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5643359ffa00 con 0x5643355d21b0 2024-09-21T05:50:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b80b1f10 con 0x5643355d21b0 2024-09-21T05:50:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f07b800b040 con 0x5643355d21b0 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 10 client.26111 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 10 client.26111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 20 client.26111 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5643355661c0 0x564335a021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 4 client.26111 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.157+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643358ea0f0 con 0x5643355661c0 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.158+0000 7f07bffff640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5643355661c0 0x564335a021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.158+0000 7f07bffff640 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5643355661c0 0x564335a021c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e16f45451aa2e08 server_cookie=eeb4744657ad76de in_seq=0 out_seq=0 2024-09-21T05:50:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.158+0000 7f079f7fe640 10 client.26111 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:45 smithi028 ceph-mon[31227]: pgmap v437: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 5.64k op/s 2024-09-21T05:50:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:45 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:46.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:45 smithi181 ceph-mon[34819]: pgmap v437: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 5.64k op/s 2024-09-21T05:50:46.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:45 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:46.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:50:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:45 smithi129 ceph-mon[34885]: pgmap v437: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 5.64k op/s 2024-09-21T05:50:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:45 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.626+0000 7f079f7fe640 1 -- 172.21.15.28:0/2081978897 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643358ea0f0 con 0x5643355661c0 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.626+0000 7f079f7fe640 10 client.26111 handle_command_reply: tid=0 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.626+0000 7f07cd1f4740 1 client.26111 shutdown 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.626+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 msgr2=0x5643359f7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.626+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359f7f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f07b80ad130 tx=0x7f07b80ad0a0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 shutdown_connections 2024-09-21T05:50:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5643355661c0 0x564335a021c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5643355d21b0 0x5643359f7f70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643355b14d0 0x564335987480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 --2- 172.21.15.28:0/2081978897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5643355a92f0 0x564335986ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 >> 172.21.15.28:0/2081978897 conn(0x5643355c6300 msgr2=0x564335979170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f079f7fe640 0 client.26111 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 shutdown_connections 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 1 -- 172.21.15.28:0/2081978897 wait complete. 2024-09-21T05:50:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.627+0000 7f07cd1f4740 20 client.26111 trim_cache size 0 max 16384 2024-09-21T05:50:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:50:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:50:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:50:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 msgr2=0x7f07a80b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 0x7f07a80b8cc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f07b00095f0 tx=0x7f07b002a880 comp rx=0 tx=0).stop 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 msgr2=0x7f07c006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fd0 tx=0x7f07b800c800 comp rx=0 tx=0).stop 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 shutdown_connections 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07c0107b70 0x7f07c006e780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07c0107220 0x7f07c006e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f07a80b67b0 0x7f07a80b8cc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 --2- 172.21.15.28:0/4094992652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c0102060 0x7f07c0074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 >> 172.21.15.28:0/4094992652 conn(0x7f07c0079e90 msgr2=0x7f07c0100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 shutdown_connections 2024-09-21T05:50:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:46.628+0000 7f07c628c640 1 -- 172.21.15.28:0/4094992652 wait complete. 2024-09-21T05:50:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.355+0000 7fbb76ff8640 1 -- 172.21.15.28:0/2810034760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70102a60 msgr2=0x7fbb70102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.355+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/2810034760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70102a60 0x7fbb70102e70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009ca0 tx=0x7fbb6402a8b0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 -- 172.21.15.28:0/2810034760 shutdown_connections 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/2810034760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb70108cf0 0x7fbb7006ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/2810034760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb701087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/2810034760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70102a60 0x7fbb70102e70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 -- 172.21.15.28:0/2810034760 >> 172.21.15.28:0/2810034760 conn(0x7fbb70079730 msgr2=0x7fbb70079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 -- 172.21.15.28:0/2810034760 shutdown_connections 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.356+0000 7fbb76ff8640 1 -- 172.21.15.28:0/2810034760 wait complete. 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.357+0000 7fbb76ff8640 1 Processor -- start 2024-09-21T05:50:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.357+0000 7fbb76ff8640 1 -- start start 2024-09-21T05:50:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.358+0000 7fbb76ff8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb70102a60 0x7fbb70119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.358+0000 7fbb76ff8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.358+0000 7fbb76ff8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70108cf0 0x7fbb70112d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb76ff8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7011a0c0 con 0x7fbb701083a0 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb76ff8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70119f40 con 0x7fbb70108cf0 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb76ff8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7011a240 con 0x7fbb70102a60 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43466/0 (socket says 172.21.15.28:43466) 2024-09-21T05:50:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb6ffff640 1 -- 172.21.15.28:0/1975267518 learned_addr learned my addr 172.21.15.28:0/1975267518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb6ffff640 1 -- 172.21.15.28:0/1975267518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb70102a60 msgr2=0x7fbb70119570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.359+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb70102a60 0x7fbb70119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.360+0000 7fbb6ffff640 1 -- 172.21.15.28:0/1975267518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70108cf0 msgr2=0x7fbb70112d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.360+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70108cf0 0x7fbb70112d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.360+0000 7fbb6ffff640 1 -- 172.21.15.28:0/1975267518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb701133c0 con 0x7fbb701083a0 2024-09-21T05:50:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.360+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00e9c0 tx=0x7fbb5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.361+0000 7fbb6dffb640 1 -- 172.21.15.28:0/1975267518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00cd80 con 0x7fbb701083a0 2024-09-21T05:50:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.361+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb701181e0 con 0x7fbb701083a0 2024-09-21T05:50:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.361+0000 7fbb6dffb640 1 -- 172.21.15.28:0/1975267518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb5c0054d0 con 0x7fbb701083a0 2024-09-21T05:50:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.361+0000 7fbb6dffb640 1 -- 172.21.15.28:0/1975267518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00cd80 con 0x7fbb701083a0 2024-09-21T05:50:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.361+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb70118770 con 0x7fbb701083a0 2024-09-21T05:50:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.365+0000 7fbb7d56c740 1 Processor -- start 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 -- start start 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x56117606d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x56117606dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561175fe3ea0 con 0x561175ca5390 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561175fd32d0 con 0x561175cc6070 2024-09-21T05:50:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.366+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561175fd70e0 con 0x561175c9d1b0 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb75595640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x56117606d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46646/0 (socket says 172.21.15.28:46646) 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb75595640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x56117606d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37016/0 (socket says 172.21.15.28:37016) 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 -- 172.21.15.28:0/1297273977 learned_addr learned my addr 172.21.15.28:0/1297273977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 -- 172.21.15.28:0/1297273977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 msgr2=0x56117606d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x56117606d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 -- 172.21.15.28:0/1297273977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 msgr2=0x56117606dd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x56117606dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.367+0000 7fbb74d94640 1 -- 172.21.15.28:0/1297273977 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561175d31650 con 0x561175c9d1b0 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.368+0000 7fbb6dffb640 1 -- 172.21.15.28:0/1975267518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbb5c021440 con 0x7fbb701083a0 2024-09-21T05:50:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.368+0000 7fbb74d94640 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009dc0 tx=0x7fbb64009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=220eebdbfb0267d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.369+0000 7fbb5affd640 1 -- 172.21.15.28:0/1297273977 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64030800 con 0x561175c9d1b0 2024-09-21T05:50:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.369+0000 7fbb5affd640 1 -- 172.21.15.28:0/1297273977 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6403f070 con 0x561175c9d1b0 2024-09-21T05:50:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.369+0000 7fbb6dffb640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 0x7fbb600b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.369+0000 7fbb75595640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 0x7fbb600b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.369+0000 7fbb6dffb640 1 -- 172.21.15.28:0/1975267518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb5c00cb80 con 0x7fbb701083a0 2024-09-21T05:50:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb5affd640 1 -- 172.21.15.28:0/1297273977 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64004330 con 0x561175c9d1b0 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 -- 172.21.15.28:0/1297273977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 msgr2=0x56117606d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009dc0 tx=0x7fbb64009400 comp rx=0 tx=0).stop 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb75595640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 0x7fbb600b8fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005e00 tx=0x7fbb6800b260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 -- 172.21.15.28:0/1297273977 shutdown_connections 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x56117606dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x56117606d800 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/1297273977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x56117606d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 -- 172.21.15.28:0/1297273977 >> 172.21.15.28:0/1297273977 conn(0x561175cba460 msgr2=0x56117606b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.370+0000 7fbb7d56c740 1 -- 172.21.15.28:0/1297273977 shutdown_connections 2024-09-21T05:50:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.371+0000 7fbb7d56c740 1 -- 172.21.15.28:0/1297273977 wait complete. 2024-09-21T05:50:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.371+0000 7fbb7d56c740 1 -- start start 2024-09-21T05:50:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.371+0000 7fbb7d56c740 1 Processor -- start 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.372+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176078980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x561176078ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x561176084e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561175e34700 con 0x561175ca5390 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561175f6e160 con 0x561175cc6070 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56117607ba90 con 0x561175c9d1b0 2024-09-21T05:50:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb7d56c740 10 client.0 _ll_register_callbacks cb 0x561175f879e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb74d94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb74d94640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176078980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46650/0 (socket says 172.21.15.28:46650) 2024-09-21T05:50:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb74d94640 1 -- 172.21.15.28:0/3458977271 learned_addr learned my addr 172.21.15.28:0/3458977271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb75595640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x561176084e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x561176078ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb75595640 1 -- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 msgr2=0x561176078980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb75595640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176078980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb75595640 1 -- 172.21.15.28:0/3458977271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 msgr2=0x561176078ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.373+0000 7fbb75595640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x561176078ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.374+0000 7fbb75595640 1 -- 172.21.15.28:0/3458977271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561175f9bc90 con 0x561175cc6070 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.374+0000 7fbb4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.374+0000 7fbb75595640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x561176084e00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005fd0 tx=0x7fbb68004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5f542b27a25cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.374+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6801e620 con 0x561175cc6070 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.374+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb6801e420 con 0x561175cc6070 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.375+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6403f070 con 0x561175cc6070 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.375+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbb6801e620 con 0x561175cc6070 2024-09-21T05:50:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.375+0000 7fbb7d56c740 10 client.26120 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.375+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561176087880 con 0x561175cc6070 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6403f070 con 0x561175cc6070 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fbb68030660 con 0x561175cc6070 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 10 client.26120 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 10 client.26120 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 20 client.26120 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175c849f0 0x56117608ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 4 client.26120 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561175fe5df0 con 0x561175c849f0 2024-09-21T05:50:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.376+0000 7fbb74d94640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175c849f0 0x56117608ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.378+0000 7fbb74d94640 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175c849f0 0x56117608ae50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b258d257017579d server_cookie=faf28ae8a73535e6 in_seq=0 out_seq=0 2024-09-21T05:50:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.378+0000 7fbb4ffff640 10 client.26120 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:50:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.379+0000 7fbb4ffff640 1 -- 172.21.15.28:0/3458977271 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561175fe5df0 con 0x561175c849f0 2024-09-21T05:50:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.379+0000 7fbb4ffff640 10 client.26120 handle_command_reply: tid=0 2024-09-21T05:50:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.379+0000 7fbb7d56c740 1 client.26120 shutdown 2024-09-21T05:50:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 msgr2=0x561176084e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x561176084e00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005fd0 tx=0x7fbb68004730 comp rx=0 tx=0).stop 2024-09-21T05:50:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 shutdown_connections 2024-09-21T05:50:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175c849f0 0x56117608ae50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175cc6070 0x561176084e00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb4ffff640 0 client.26120 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.380+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x561176078ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.381+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/3458977271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.381+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 >> 172.21.15.28:0/3458977271 conn(0x561175cba460 msgr2=0x561176070450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.381+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 shutdown_connections 2024-09-21T05:50:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.381+0000 7fbb7d56c740 1 -- 172.21.15.28:0/3458977271 wait complete. 2024-09-21T05:50:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.381+0000 7fbb7d56c740 20 client.26120 trim_cache size 0 max 16384 2024-09-21T05:50:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.405+0000 7fbb7d56c740 1 Processor -- start 2024-09-21T05:50:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.405+0000 7fbb7d56c740 1 -- start start 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 0x56117606ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176075b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x5611760760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701be640 con 0x561175ca5390 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701c02d0 con 0x561175c849f0 2024-09-21T05:50:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701bed10 con 0x561175c9d1b0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb75595640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x5611760760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176075b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb75595640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x5611760760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43494/0 (socket says 172.21.15.28:43494) 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb75595640 1 -- 172.21.15.28:0/492990543 learned_addr learned my addr 172.21.15.28:0/492990543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 0x56117606ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 -- 172.21.15.28:0/492990543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 msgr2=0x561176075b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176075b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 -- 172.21.15.28:0/492990543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 msgr2=0x5611760760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x5611760760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.406+0000 7fbb74d94640 1 -- 172.21.15.28:0/492990543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561176076e80 con 0x561175c849f0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.407+0000 7fbb74d94640 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 0x56117606ea00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009960 tx=0x7fbb64003ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd34f5474b0f410 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.407+0000 7fbb5a7fc640 1 -- 172.21.15.28:0/492990543 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64030a00 con 0x561175c849f0 2024-09-21T05:50:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.407+0000 7fbb5a7fc640 1 -- 172.21.15.28:0/492990543 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6403f040 con 0x561175c849f0 2024-09-21T05:50:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.407+0000 7fbb5a7fc640 1 -- 172.21.15.28:0/492990543 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64004850 con 0x561175c849f0 2024-09-21T05:50:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 -- 172.21.15.28:0/492990543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 msgr2=0x56117606ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 0x56117606ea00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009960 tx=0x7fbb64003ee0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 -- 172.21.15.28:0/492990543 shutdown_connections 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175ca5390 0x5611760760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175c9d1b0 0x561176075b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/492990543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c849f0 0x56117606ea00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 -- 172.21.15.28:0/492990543 >> 172.21.15.28:0/492990543 conn(0x561175cba460 msgr2=0x56117606e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 -- 172.21.15.28:0/492990543 shutdown_connections 2024-09-21T05:50:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.409+0000 7fbb7d56c740 1 -- 172.21.15.28:0/492990543 wait complete. 2024-09-21T05:50:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.410+0000 7fbb7d56c740 1 -- start start 2024-09-21T05:50:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.410+0000 7fbb7d56c740 1 Processor -- start 2024-09-21T05:50:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.411+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 0x56117607f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.411+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb7d56c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175ca5390 0x5611760f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701bfb40 con 0x561175c849f0 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701bf440 con 0x561175c9d1b0 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb7d56c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701bc5e0 con 0x561175ca5390 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37038/0 (socket says 172.21.15.28:37038) 2024-09-21T05:50:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb6ffff640 1 -- 172.21.15.28:0/605958647 learned_addr learned my addr 172.21.15.28:0/605958647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb75595640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175ca5390 0x5611760f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb74d94640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 0x56117607f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.412+0000 7fbb7d56c740 10 client.0 _ll_register_callbacks cb 0x7fbb68005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 -- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175ca5390 msgr2=0x5611760f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175ca5390 0x5611760f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 -- 172.21.15.28:0/605958647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 msgr2=0x56117607f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 0x56117607f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 -- 172.21.15.28:0/605958647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611760f4da0 con 0x561175c9d1b0 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb74d94640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 0x56117607f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb6ffff640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0ad1b0 tx=0x7fbb5c0acf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71b26b1eaa654f2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0adbb0 con 0x561175c9d1b0 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbb5c0ac710 con 0x561175c9d1b0 2024-09-21T05:50:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.413+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6403f040 con 0x561175c9d1b0 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbb5c0b1f90 con 0x561175c9d1b0 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 10 client.26126 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5611760f5f30 con 0x561175c9d1b0 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6403f040 con 0x561175c9d1b0 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fbb5c0ac310 con 0x561175c9d1b0 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 10 client.26126 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 10 client.26126 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:50:47.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 20 client.26126 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175cc6070 0x5611760f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 4 client.26126 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:50:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.414+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561175fe18a0 con 0x561175cc6070 2024-09-21T05:50:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.415+0000 7fbb74d94640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175cc6070 0x5611760f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:47.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.417+0000 7fbb4ffff640 10 client.26126 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:50:47.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.417+0000 7fbb74d94640 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175cc6070 0x5611760f87a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef070a8db2733cb8 server_cookie=ab0b83fb55e095d7 in_seq=0 out_seq=0 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.418+0000 7fbb4ffff640 1 -- 172.21.15.28:0/605958647 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561175fe18a0 con 0x561175cc6070 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.418+0000 7fbb4ffff640 10 client.26126 handle_command_reply: tid=0 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.418+0000 7fbb7d56c740 1 client.26126 shutdown 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 msgr2=0x56117607f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0ad1b0 tx=0x7fbb5c0acf30 comp rx=0 tx=0).stop 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 shutdown_connections 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561175cc6070 0x5611760f87a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb4ffff640 0 client.26126 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561175ca5390 0x5611760f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561175c9d1b0 0x56117607f770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 --2- 172.21.15.28:0/605958647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561175c849f0 0x56117607f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 >> 172.21.15.28:0/605958647 conn(0x561175cba460 msgr2=0x56117606e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 shutdown_connections 2024-09-21T05:50:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.419+0000 7fbb7d56c740 1 -- 172.21.15.28:0/605958647 wait complete. 2024-09-21T05:50:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.420+0000 7fbb7d56c740 20 client.26126 trim_cache size 0 max 16384 2024-09-21T05:50:47.422 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:50:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.420+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 msgr2=0x7fbb600b8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.420+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 0x7fbb600b8fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbb68005e00 tx=0x7fbb6800b260 comp rx=0 tx=0).stop 2024-09-21T05:50:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.420+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 msgr2=0x7fbb70112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.420+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00e9c0 tx=0x7fbb5c00ee80 comp rx=0 tx=0).stop 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 shutdown_connections 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb70108cf0 0x7fbb70112d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbb600b6ac0 0x7fbb600b8fd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb701083a0 0x7fbb70112840 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 --2- 172.21.15.28:0/1975267518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb70102a60 0x7fbb70119570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 >> 172.21.15.28:0/1975267518 conn(0x7fbb70079730 msgr2=0x7fbb70106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 shutdown_connections 2024-09-21T05:50:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:47.421+0000 7fbb76ff8640 1 -- 172.21.15.28:0/1975267518 wait complete. 2024-09-21T05:50:48.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:47 smithi181 ceph-mon[34819]: pgmap v438: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.34k op/s 2024-09-21T05:50:48.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:47 smithi028 ceph-mon[31227]: pgmap v438: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.34k op/s 2024-09-21T05:50:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:47 smithi129 ceph-mon[34885]: pgmap v438: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.34k op/s 2024-09-21T05:50:48.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:50:48.973 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:50.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 -- 172.21.15.28:0/1718488195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 msgr2=0x7f896c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 --2- 172.21.15.28:0/1718488195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 0x7f896c10f520 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8964009ca0 tx=0x7f896402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 -- 172.21.15.28:0/1718488195 shutdown_connections 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 --2- 172.21.15.28:0/1718488195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 0x7f896c10f520 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 --2- 172.21.15.28:0/1718488195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c1083a0 0x7f896c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 --2- 172.21.15.28:0/1718488195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c102a60 0x7f896c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 -- 172.21.15.28:0/1718488195 >> 172.21.15.28:0/1718488195 conn(0x7f896c079730 msgr2=0x7f896c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.033+0000 7f8971c82640 1 -- 172.21.15.28:0/1718488195 shutdown_connections 2024-09-21T05:50:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.034+0000 7f8971c82640 1 -- 172.21.15.28:0/1718488195 wait complete. 2024-09-21T05:50:50.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 Processor -- start 2024-09-21T05:50:50.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 -- start start 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c1083a0 0x7f896c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 0x7f896c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1118a0 con 0x7f896c108cf0 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f896c111720 con 0x7f896c1083a0 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f8971c82640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f896c111a20 con 0x7f896c102a60 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46688/0 (socket says 172.21.15.28:46688) 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.035+0000 7f896b7fe640 1 -- 172.21.15.28:0/1584207625 learned_addr learned my addr 172.21.15.28:0/1584207625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.036+0000 7f8963fff640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c1083a0 0x7f896c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.036+0000 7f896b7fe640 1 -- 172.21.15.28:0/1584207625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c1083a0 msgr2=0x7f896c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.036+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c1083a0 0x7f896c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.036+0000 7f896b7fe640 1 -- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 msgr2=0x7f896c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.037+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 0x7f896c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.037+0000 7f896b7fe640 1 -- 172.21.15.28:0/1584207625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c1acb70 con 0x7f896c102a60 2024-09-21T05:50:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.037+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f895c00d950 tx=0x7f895c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.038+0000 7f89697fa640 1 -- 172.21.15.28:0/1584207625 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c017070 con 0x7f896c102a60 2024-09-21T05:50:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.038+0000 7f89697fa640 1 -- 172.21.15.28:0/1584207625 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895c009c30 con 0x7f896c102a60 2024-09-21T05:50:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.038+0000 7f89697fa640 1 -- 172.21.15.28:0/1584207625 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c00f620 con 0x7f896c102a60 2024-09-21T05:50:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.040+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c19c370 con 0x7f896c102a60 2024-09-21T05:50:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.040+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c19c830 con 0x7f896c102a60 2024-09-21T05:50:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.044+0000 7f89781f6740 1 Processor -- start 2024-09-21T05:50:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.045+0000 7f89781f6740 1 -- start start 2024-09-21T05:50:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.045+0000 7f89697fa640 1 -- 172.21.15.28:0/1584207625 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f895c010460 con 0x7f896c102a60 2024-09-21T05:50:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.045+0000 7f89781f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefd0070 0x5556ef3776a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.046+0000 7f89781f6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef377be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.046+0000 7f89781f6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.046+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef377be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.047+0000 7f89697fa640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 0x7f89580b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.047+0000 7f8963fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.047+0000 7f8963fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46704/0 (socket says 172.21.15.28:46704) 2024-09-21T05:50:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.047+0000 7f8963fff640 1 -- 172.21.15.28:0/948239401 learned_addr learned my addr 172.21.15.28:0/948239401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.047+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556ef2edea0 con 0x5556eefd0070 2024-09-21T05:50:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.048+0000 7f89697fa640 1 -- 172.21.15.28:0/1584207625 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f895c0e70d0 con 0x7f896c102a60 2024-09-21T05:50:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.048+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556ef2dd2d0 con 0x5556eefa71b0 2024-09-21T05:50:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.048+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5556ef2e10e0 con 0x5556eefaf390 2024-09-21T05:50:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.049+0000 7f896bfff640 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefd0070 0x5556ef3776a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 -- 172.21.15.28:0/948239401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 msgr2=0x5556ef377be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef377be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 -- 172.21.15.28:0/948239401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefd0070 msgr2=0x5556ef3776a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefd0070 0x5556ef3776a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 -- 172.21.15.28:0/948239401 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556ef03b650 con 0x5556eefaf390 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f896bfff640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 0x7f89580b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.050+0000 7f8963fff640 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89540044d0 tx=0x7f8954005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f53bff6e98d37e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.051+0000 7f8961ffb640 1 -- 172.21.15.28:0/948239401 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954003b20 con 0x5556eefaf390 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.051+0000 7f8961ffb640 1 -- 172.21.15.28:0/948239401 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895402c070 con 0x5556eefaf390 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 msgr2=0x5556ef378120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f896bfff640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 0x7f89580b8b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f89640094e0 tx=0x7f8964005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f8961ffb640 1 -- 172.21.15.28:0/948239401 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954003b20 con 0x5556eefaf390 2024-09-21T05:50:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89540044d0 tx=0x7f8954005f90 comp rx=0 tx=0).stop 2024-09-21T05:50:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 shutdown_connections 2024-09-21T05:50:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef378120 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef377be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 --2- 172.21.15.28:0/948239401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefd0070 0x5556ef3776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.052+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 >> 172.21.15.28:0/948239401 conn(0x5556eefc4460 msgr2=0x5556ef375590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.053+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 shutdown_connections 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.053+0000 7f89781f6740 1 -- 172.21.15.28:0/948239401 wait complete. 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.054+0000 7f89781f6740 1 -- start start 2024-09-21T05:50:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 Processor -- start 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef382c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefaf390 0x5556ef383140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556ef13e700 con 0x5556eefa71b0 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556ef278160 con 0x5556eefaf390 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.055+0000 7f89781f6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5556ef385ed0 con 0x5556eefd0070 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f89781f6740 10 client.0 _ll_register_callbacks cb 0x5556ef2919e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f8963fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefaf390 0x5556ef383140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46712/0 (socket says 172.21.15.28:46712) 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 -- 172.21.15.28:0/603947555 learned_addr learned my addr 172.21.15.28:0/603947555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896b7fe640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef382c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 -- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefaf390 msgr2=0x5556ef383140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefaf390 0x5556ef383140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 -- 172.21.15.28:0/603947555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 msgr2=0x5556ef382c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef382c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.056+0000 7f896bfff640 1 -- 172.21.15.28:0/603947555 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556ef2a5c90 con 0x5556eefd0070 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f896bfff640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89640316d0 tx=0x7f89640095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315b50937ea17b10 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f894affd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896404e060 con 0x5556eefd0070 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f89640044c0 con 0x5556eefd0070 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f895402c070 con 0x5556eefd0070 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.057+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8964031910 con 0x5556eefd0070 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 10 client.26155 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5556ef391ad0 con 0x5556eefd0070 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8964036040 con 0x5556eefd0070 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f89640305c0 con 0x5556eefd0070 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 10 client.26155 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 10 client.26155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 20 client.26155 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eef8e9f0 0x5556ef395050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.058+0000 7f89781f6740 4 client.26155 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.059+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f895402c070 con 0x5556eef8e9f0 2024-09-21T05:50:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.059+0000 7f896b7fe640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eef8e9f0 0x5556ef395050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.060+0000 7f896b7fe640 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eef8e9f0 0x5556ef395050 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d15e1e13ae9c1362 server_cookie=ad1afe0ed9d46ca in_seq=0 out_seq=0 2024-09-21T05:50:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.060+0000 7f894affd640 10 client.26155 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:50:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.061+0000 7f894affd640 1 -- 172.21.15.28:0/603947555 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f895402c070 con 0x5556eef8e9f0 2024-09-21T05:50:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.061+0000 7f894affd640 10 client.26155 handle_command_reply: tid=0 2024-09-21T05:50:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.061+0000 7f89781f6740 1 client.26155 shutdown 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.061+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 msgr2=0x5556ef383680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89640316d0 tx=0x7f89640095c0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 shutdown_connections 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eef8e9f0 0x5556ef395050 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefd0070 0x5556ef383680 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f894affd640 0 client.26155 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefaf390 0x5556ef383140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 --2- 172.21.15.28:0/603947555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef382c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.062+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 >> 172.21.15.28:0/603947555 conn(0x5556eefc4460 msgr2=0x5556ef37a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.064+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 shutdown_connections 2024-09-21T05:50:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.064+0000 7f89781f6740 1 -- 172.21.15.28:0/603947555 wait complete. 2024-09-21T05:50:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.064+0000 7f89781f6740 20 client.26155 trim_cache size 0 max 16384 2024-09-21T05:50:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.091+0000 7f89781f6740 1 Processor -- start 2024-09-21T05:50:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.091+0000 7f89781f6740 1 -- start start 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.091+0000 7f89781f6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eef8e9f0 0x5556ef378de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.091+0000 7f89781f6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.091+0000 7f89781f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefaf390 0x5556ef39ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f89781f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1ad4d0 con 0x5556eefaf390 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f89781f6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1af160 con 0x5556eefa71b0 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f89781f6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1adba0 con 0x5556eef8e9f0 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37070/0 (socket says 172.21.15.28:37070) 2024-09-21T05:50:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 -- 172.21.15.28:0/2906940505 learned_addr learned my addr 172.21.15.28:0/2906940505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f896bfff640 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefaf390 0x5556ef39ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 -- 172.21.15.28:0/2906940505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eef8e9f0 msgr2=0x5556ef378de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eef8e9f0 0x5556ef378de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 -- 172.21.15.28:0/2906940505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefaf390 msgr2=0x5556ef39ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefaf390 0x5556ef39ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.092+0000 7f8963fff640 1 -- 172.21.15.28:0/2906940505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556ef3784b0 con 0x5556eefa71b0 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.093+0000 7f8963fff640 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f89540044a0 tx=0x7f8954027cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4882587928f4cef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.093+0000 7f89617fa640 1 -- 172.21.15.28:0/2906940505 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954027800 con 0x5556eefa71b0 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.093+0000 7f89617fa640 1 -- 172.21.15.28:0/2906940505 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895402c030 con 0x5556eefa71b0 2024-09-21T05:50:50.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.093+0000 7f89617fa640 1 -- 172.21.15.28:0/2906940505 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954003ee0 con 0x5556eefa71b0 2024-09-21T05:50:50.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 -- 172.21.15.28:0/2906940505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 msgr2=0x5556ef39a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f89540044a0 tx=0x7f8954027cc0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 -- 172.21.15.28:0/2906940505 shutdown_connections 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefaf390 0x5556ef39ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eefa71b0 0x5556ef39a810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 --2- 172.21.15.28:0/2906940505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eef8e9f0 0x5556ef378de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.095+0000 7f89781f6740 1 -- 172.21.15.28:0/2906940505 >> 172.21.15.28:0/2906940505 conn(0x5556eefc4460 msgr2=0x5556ef3786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.096+0000 7f89781f6740 1 -- 172.21.15.28:0/2906940505 shutdown_connections 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.096+0000 7f89781f6740 1 -- 172.21.15.28:0/2906940505 wait complete. 2024-09-21T05:50:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.096+0000 7f89781f6740 1 -- start start 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 Processor -- start 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef387850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef387d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1ae9d0 con 0x5556eefa71b0 2024-09-21T05:50:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1ae2d0 con 0x5556eef8e9f0 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1a3280 con 0x5556eefaf390 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.097+0000 7f89781f6740 10 client.0 _ll_register_callbacks cb 0x7f8954005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef387850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f896b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37078/0 (socket says 172.21.15.28:37078) 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f896b7fe640 1 -- 172.21.15.28:0/1225481000 learned_addr learned my addr 172.21.15.28:0/1225481000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f896bfff640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef387d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 -- 172.21.15.28:0/1225481000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 msgr2=0x5556ef387d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef387d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 -- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 msgr2=0x5556ef387310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f8963fff640 1 -- 172.21.15.28:0/1225481000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556ef3feb90 con 0x5556eefa71b0 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.098+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.099+0000 7f8963fff640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef387850 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8954000c00 tx=0x7f8954004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb06f5311fa6865 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.099+0000 7f894affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.099+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954027810 con 0x5556eefa71b0 2024-09-21T05:50:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.099+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f895402c030 con 0x5556eefa71b0 2024-09-21T05:50:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.099+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8954040040 con 0x5556eefa71b0 2024-09-21T05:50:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.100+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f89540279b0 con 0x5556eefa71b0 2024-09-21T05:50:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.100+0000 7f89781f6740 10 client.16554 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.100+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5556ef3ffe10 con 0x5556eefa71b0 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895402c030 con 0x5556eefa71b0 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f8954044070 con 0x5556eefa71b0 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 10 client.16554 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 10 client.16554 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 20 client.16554 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eefd0070 0x5556ef402310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 4 client.16554 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556ef2efdf0 con 0x5556eefd0070 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.101+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eefd0070 0x5556ef402310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.102+0000 7f894affd640 10 client.16554 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.102+0000 7f896b7fe640 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eefd0070 0x5556ef402310 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0ab36f30e9ff3d5 server_cookie=6048e27120f2f2ac in_seq=0 out_seq=0 2024-09-21T05:50:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f894affd640 1 -- 172.21.15.28:0/1225481000 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556ef2efdf0 con 0x5556eefd0070 2024-09-21T05:50:50.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f894affd640 10 client.16554 handle_command_reply: tid=0 2024-09-21T05:50:50.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 client.16554 shutdown 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 msgr2=0x5556ef387850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef387850 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8954000c00 tx=0x7f8954004040 comp rx=0 tx=0).stop 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 shutdown_connections 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5556eefd0070 0x5556ef402310 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556eefaf390 0x5556ef387d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556eefa71b0 0x5556ef387850 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f894affd640 0 client.16554 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 --2- 172.21.15.28:0/1225481000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556eef8e9f0 0x5556ef387310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 >> 172.21.15.28:0/1225481000 conn(0x5556eefc4460 msgr2=0x5556ef3840b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 shutdown_connections 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 1 -- 172.21.15.28:0/1225481000 wait complete. 2024-09-21T05:50:50.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.103+0000 7f89781f6740 20 client.16554 trim_cache size 0 max 16384 2024-09-21T05:50:50.107 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:50:50.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 msgr2=0x7f89580b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 0x7f89580b8b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f89640094e0 tx=0x7f8964005fd0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 msgr2=0x7f896c19bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f895c00d950 tx=0x7f895c00de10 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 shutdown_connections 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f89580b6680 0x7f89580b8b90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c108cf0 0x7f896c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f896c1083a0 0x7f896c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 --2- 172.21.15.28:0/1584207625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f896c102a60 0x7f896c19bd40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.104+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 >> 172.21.15.28:0/1584207625 conn(0x7f896c079730 msgr2=0x7f896c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.105+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 shutdown_connections 2024-09-21T05:50:50.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:50.105+0000 7f8971c82640 1 -- 172.21.15.28:0/1584207625 wait complete. 2024-09-21T05:50:50.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:49 smithi181 ceph-mon[34819]: pgmap v439: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 25 MiB/s wr, 4.60k op/s 2024-09-21T05:50:50.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:49 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:50:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:49 smithi028 ceph-mon[31227]: pgmap v439: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 25 MiB/s wr, 4.60k op/s 2024-09-21T05:50:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:49 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:50:50.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:49 smithi129 ceph-mon[34885]: pgmap v439: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 25 MiB/s wr, 4.60k op/s 2024-09-21T05:50:50.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:49 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:50:50.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:50:51.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:50:51.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:50:51.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:50 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:50:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:50:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:50:51.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:50 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:50:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:51.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:50:51.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:50:51.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:50 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 -- 172.21.15.28:0/1554402307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5081083a0 msgr2=0x7fc5081087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1554402307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5081083a0 0x7fc5081087b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009900 tx=0x7fc4fc02a410 comp rx=0 tx=0).stop 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 -- 172.21.15.28:0/1554402307 shutdown_connections 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1554402307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc508108cf0 0x7fc50806ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1554402307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5081083a0 0x7fc5081087b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1554402307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc508102a60 0x7fc508102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.948+0000 7fc5105aa640 1 -- 172.21.15.28:0/1554402307 >> 172.21.15.28:0/1554402307 conn(0x7fc508079730 msgr2=0x7fc508079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.949+0000 7fc5105aa640 1 -- 172.21.15.28:0/1554402307 shutdown_connections 2024-09-21T05:50:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.949+0000 7fc5105aa640 1 -- 172.21.15.28:0/1554402307 wait complete. 2024-09-21T05:50:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 Processor -- start 2024-09-21T05:50:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 -- start start 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5081083a0 0x7fc508112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc508108cf0 0x7fc5081134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc508115cc0 con 0x7fc508102a60 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc508115b40 con 0x7fc508108cf0 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.950+0000 7fc5105aa640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc508115e40 con 0x7fc5081083a0 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43558/0 (socket says 172.21.15.28:43558) 2024-09-21T05:50:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 -- 172.21.15.28:0/1529439365 learned_addr learned my addr 172.21.15.28:0/1529439365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:51.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 -- 172.21.15.28:0/1529439365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5081083a0 msgr2=0x7fc508112f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5081083a0 0x7fc508112f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 -- 172.21.15.28:0/1529439365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc508108cf0 msgr2=0x7fc5081134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc508108cf0 0x7fc5081134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.951+0000 7fc50e346640 1 -- 172.21.15.28:0/1529439365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5081acb40 con 0x7fc508102a60 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.952+0000 7fc50e346640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc50400e9f0 tx=0x7fc50400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.952+0000 7fc4fb7fe640 1 -- 172.21.15.28:0/1529439365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50400cd80 con 0x7fc508102a60 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.953+0000 7fc4fb7fe640 1 -- 172.21.15.28:0/1529439365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5040108e0 con 0x7fc508102a60 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.953+0000 7fc4fb7fe640 1 -- 172.21.15.28:0/1529439365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504018980 con 0x7fc508102a60 2024-09-21T05:50:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.953+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5081ace30 con 0x7fc508102a60 2024-09-21T05:50:51.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.953+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5081ad290 con 0x7fc508102a60 2024-09-21T05:50:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.957+0000 7fc516b1e740 1 Processor -- start 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.957+0000 7fc516b1e740 1 -- start start 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 0x556a04247800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04247d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a041bdea0 con 0x556a03e771b0 2024-09-21T05:50:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556a041ad2d0 con 0x556a03ea0070 2024-09-21T05:50:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.958+0000 7fc516b1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556a041b10e0 con 0x556a03e7f390 2024-09-21T05:50:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.959+0000 7fc50db45640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04247d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.959+0000 7fc50db45640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04247d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46776/0 (socket says 172.21.15.28:46776) 2024-09-21T05:50:51.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.959+0000 7fc50db45640 1 -- 172.21.15.28:0/2038162342 learned_addr learned my addr 172.21.15.28:0/2038162342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:51.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.960+0000 7fc50eb47640 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.960+0000 7fc50e346640 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 0x556a04247800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.960+0000 7fc4fb7fe640 1 -- 172.21.15.28:0/1529439365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc504018b20 con 0x7fc508102a60 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.961+0000 7fc50e346640 1 -- 172.21.15.28:0/2038162342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 msgr2=0x556a04247d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.961+0000 7fc50e346640 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04247d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.961+0000 7fc50e346640 1 -- 172.21.15.28:0/2038162342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 msgr2=0x556a042472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.961+0000 7fc50e346640 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.961+0000 7fc50e346640 1 -- 172.21.15.28:0/2038162342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a03f0b650 con 0x556a03e771b0 2024-09-21T05:50:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.962+0000 7fc4fb7fe640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 0x7fc4f40b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.962+0000 7fc50e346640 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 0x556a04247800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc5040e5160 tx=0x7fc5040f0180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5098815156436f54 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.962+0000 7fc4fb7fe640 1 -- 172.21.15.28:0/1529439365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc5040e7a50 con 0x7fc508102a60 2024-09-21T05:50:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.962+0000 7fc4f8ff9640 1 -- 172.21.15.28:0/2038162342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040a9eb0 con 0x556a03e771b0 2024-09-21T05:50:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.963+0000 7fc4f8ff9640 1 -- 172.21.15.28:0/2038162342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5040f1040 con 0x556a03e771b0 2024-09-21T05:50:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.963+0000 7fc4f8ff9640 1 -- 172.21.15.28:0/2038162342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040acfd0 con 0x556a03e771b0 2024-09-21T05:50:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.963+0000 7fc50eb47640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 0x7fc4f40b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc50eb47640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 0x7fc4f40b8d60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc500009270 tx=0x7fc5000095e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 -- 172.21.15.28:0/2038162342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 msgr2=0x556a04247800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 0x556a04247800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc5040e5160 tx=0x7fc5040f0180 comp rx=0 tx=0).stop 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 -- 172.21.15.28:0/2038162342 shutdown_connections 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04247d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e771b0 0x556a04247800 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 --2- 172.21.15.28:0/2038162342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.964+0000 7fc516b1e740 1 -- 172.21.15.28:0/2038162342 >> 172.21.15.28:0/2038162342 conn(0x556a03e94460 msgr2=0x556a042451b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.965+0000 7fc516b1e740 1 -- 172.21.15.28:0/2038162342 shutdown_connections 2024-09-21T05:50:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.965+0000 7fc516b1e740 1 -- 172.21.15.28:0/2038162342 wait complete. 2024-09-21T05:50:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.965+0000 7fc516b1e740 1 -- start start 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 Processor -- start 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04252e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03ea0070 0x556a04253390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a0400e700 con 0x556a03ea0070 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556a04148160 con 0x556a03e771b0 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556a04255a40 con 0x556a03e7f390 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc516b1e740 10 client.0 _ll_register_callbacks cb 0x556a041619e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.966+0000 7fc50e346640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37114/0 (socket says 172.21.15.28:37114) 2024-09-21T05:50:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 -- 172.21.15.28:0/1291674605 learned_addr learned my addr 172.21.15.28:0/1291674605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50db45640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04252e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50eb47640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03ea0070 0x556a04253390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 -- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 msgr2=0x556a04252e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04252e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 -- 172.21.15.28:0/1291674605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03ea0070 msgr2=0x556a04253390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03ea0070 0x556a04253390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.967+0000 7fc50e346640 1 -- 172.21.15.28:0/1291674605 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a04175c90 con 0x556a03e771b0 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.968+0000 7fc50e346640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc5040e5130 tx=0x7fc5040e5610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e0d16d6c27e51e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.968+0000 7fc4f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:51.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040ad1b0 con 0x556a03e771b0 2024-09-21T05:50:51.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5040f1040 con 0x556a03e771b0 2024-09-21T05:50:51.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc5040aeb90 con 0x556a03e771b0 2024-09-21T05:50:51.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc5040ad630 con 0x556a03e771b0 2024-09-21T05:50:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc516b1e740 10 client.26153 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.969+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556a04261720 con 0x556a03e771b0 2024-09-21T05:50:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5040f1040 con 0x556a03e771b0 2024-09-21T05:50:51.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fc5040acac0 con 0x556a03e771b0 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 10 client.26153 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 10 client.26153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 20 client.26153 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e5e9f0 0x556a04264d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 4 client.26153 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.970+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a041bfdf0 con 0x556a03e5e9f0 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.971+0000 7fc50db45640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e5e9f0 0x556a04264d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.972+0000 7fc50db45640 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e5e9f0 0x556a04264d40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86e58316f06eff90 server_cookie=ebcb83f4cd5e062d in_seq=0 out_seq=0 2024-09-21T05:50:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.972+0000 7fc4f1ffb640 10 client.26153 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.973+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/1291674605 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556a041bfdf0 con 0x556a03e5e9f0 2024-09-21T05:50:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.973+0000 7fc4f1ffb640 10 client.26153 handle_command_reply: tid=0 2024-09-21T05:50:51.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.974+0000 7fc516b1e740 1 client.26153 shutdown 2024-09-21T05:50:51.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.975+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 msgr2=0x556a04252910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:51.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.975+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc5040e5130 tx=0x7fc5040e5610 comp rx=0 tx=0).stop 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.975+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 shutdown_connections 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.975+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e5e9f0 0x556a04264d40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03ea0070 0x556a04253390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e7f390 0x556a04252e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 --2- 172.21.15.28:0/1291674605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03e771b0 0x556a04252910 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 >> 172.21.15.28:0/1291674605 conn(0x556a03e94460 msgr2=0x556a0424a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:51.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc4f1ffb640 0 client.26153 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:51.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 shutdown_connections 2024-09-21T05:50:51.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.976+0000 7fc516b1e740 1 -- 172.21.15.28:0/1291674605 wait complete. 2024-09-21T05:50:51.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:51.977+0000 7fc516b1e740 20 client.26153 trim_cache size 0 max 16384 2024-09-21T05:50:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.002+0000 7fc516b1e740 1 Processor -- start 2024-09-21T05:50:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.002+0000 7fc516b1e740 1 -- start start 2024-09-21T05:50:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04254900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04244760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00e770 con 0x556a03e7f390 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec010400 con 0x556a03ea0070 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc516b1e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00ee40 con 0x556a03e771b0 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc50e346640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04254900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc50eb47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc50eb47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37128/0 (socket says 172.21.15.28:37128) 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc50eb47640 1 -- 172.21.15.28:0/3373603681 learned_addr learned my addr 172.21.15.28:0/3373603681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.003+0000 7fc50db45640 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04244760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 -- 172.21.15.28:0/3373603681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 msgr2=0x556a04254900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04254900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 -- 172.21.15.28:0/3373603681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 msgr2=0x556a04244ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 -- 172.21.15.28:0/3373603681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a04253bd0 con 0x556a03e7f390 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50eb47640 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.004+0000 7fc50db45640 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04244760 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005ab0 tx=0x7fc4fc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729c18028169f607 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.005+0000 7fc4f3fff640 1 -- 172.21.15.28:0/3373603681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc035040 con 0x556a03e7f390 2024-09-21T05:50:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.005+0000 7fc4f3fff640 1 -- 172.21.15.28:0/3373603681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4fc003ca0 con 0x556a03e7f390 2024-09-21T05:50:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.005+0000 7fc4f3fff640 1 -- 172.21.15.28:0/3373603681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc031bf0 con 0x556a03e7f390 2024-09-21T05:50:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 -- 172.21.15.28:0/3373603681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 msgr2=0x556a04244760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04244760 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005ab0 tx=0x7fc4fc005fd0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 -- 172.21.15.28:0/3373603681 shutdown_connections 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a04244ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04244760 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 --2- 172.21.15.28:0/3373603681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04254900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.007+0000 7fc516b1e740 1 -- 172.21.15.28:0/3373603681 >> 172.21.15.28:0/3373603681 conn(0x556a03e94460 msgr2=0x556a04248cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.008+0000 7fc516b1e740 1 -- 172.21.15.28:0/3373603681 shutdown_connections 2024-09-21T05:50:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.008+0000 7fc516b1e740 1 -- 172.21.15.28:0/3373603681 wait complete. 2024-09-21T05:50:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.008+0000 7fc516b1e740 1 -- start start 2024-09-21T05:50:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.008+0000 7fc516b1e740 1 Processor -- start 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04256af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc516b1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50db45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04256af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc516b1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00fc70 con 0x556a03e7f390 2024-09-21T05:50:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50e346640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50e346640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46806/0 (socket says 172.21.15.28:46806) 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50db45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04256af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43582/0 (socket says 172.21.15.28:43582) 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50e346640 1 -- 172.21.15.28:0/157924089 learned_addr learned my addr 172.21.15.28:0/157924089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc50eb47640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.009+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00f570 con 0x556a03ea0070 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00c750 con 0x556a03e771b0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc516b1e740 10 client.0 _ll_register_callbacks cb 0x7fc500009500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 -- 172.21.15.28:0/157924089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 msgr2=0x556a042cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 -- 172.21.15.28:0/157924089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 msgr2=0x556a04256af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04256af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a042cdd60 con 0x556a03e771b0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc50e346640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc504005eb0 tx=0x7fc5040f0790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c2b4386c846d96 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.010+0000 7fc4f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:50:52.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040a9700 con 0x556a03e771b0 2024-09-21T05:50:52.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc5040eb2d0 con 0x556a03e771b0 2024-09-21T05:50:52.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc003ca0 con 0x556a03e771b0 2024-09-21T05:50:52.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc5040ad1b0 con 0x556a03e771b0 2024-09-21T05:50:52.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc516b1e740 10 client.26173 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556a042cefa0 con 0x556a03e771b0 2024-09-21T05:50:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.012+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4fc003ca0 con 0x556a03e771b0 2024-09-21T05:50:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fc5040a9240 con 0x556a03e771b0 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 10 client.26173 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 10 client.26173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 20 client.26173 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e33da0 0x556a042d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 4 client.26173 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.013+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a041bb8a0 con 0x556a03e33da0 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.014+0000 7fc50db45640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e33da0 0x556a042d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.014+0000 7fc50db45640 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e33da0 0x556a042d15c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abfef64a2fbdcd01 server_cookie=e60fd53f9ab6e0fa in_seq=0 out_seq=0 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc4f1ffb640 10 client.26173 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc4f1ffb640 1 -- 172.21.15.28:0/157924089 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a041bb8a0 con 0x556a03e33da0 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc4f1ffb640 10 client.26173 handle_command_reply: tid=0 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc516b1e740 1 client.26173 shutdown 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 msgr2=0x556a04256530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.015+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc504005eb0 tx=0x7fc5040f0790 comp rx=0 tx=0).stop 2024-09-21T05:50:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 shutdown_connections 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556a03e33da0 0x556a042d15c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a03ea0070 0x556a042cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc4f1ffb640 0 client.26173 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a03e7f390 0x556a04256af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 --2- 172.21.15.28:0/157924089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556a03e771b0 0x556a04256530 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 >> 172.21.15.28:0/157924089 conn(0x556a03e94460 msgr2=0x556a042526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 shutdown_connections 2024-09-21T05:50:52.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 1 -- 172.21.15.28:0/157924089 wait complete. 2024-09-21T05:50:52.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.016+0000 7fc516b1e740 20 client.26173 trim_cache size 0 max 16384 2024-09-21T05:50:52.019 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 msgr2=0x7fc4f40b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 0x7fc4f40b8d60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc500009270 tx=0x7fc5000095e0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 msgr2=0x7fc508112a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc50400e9f0 tx=0x7fc50400eeb0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 shutdown_connections 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc508108cf0 0x7fc5081134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5081083a0 0x7fc508112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc4f40b6850 0x7fc4f40b8d60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 --2- 172.21.15.28:0/1529439365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc508102a60 0x7fc508112a40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.017+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 >> 172.21.15.28:0/1529439365 conn(0x7fc508079730 msgr2=0x7fc508101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.018+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 shutdown_connections 2024-09-21T05:50:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:52.018+0000 7fc5105aa640 1 -- 172.21.15.28:0/1529439365 wait complete. 2024-09-21T05:50:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:51 smithi028 ceph-mon[31227]: pgmap v440: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 24 MiB/s wr, 4.59k op/s 2024-09-21T05:50:52.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:51 smithi181 ceph-mon[34819]: pgmap v440: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 24 MiB/s wr, 4.59k op/s 2024-09-21T05:50:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:51 smithi129 ceph-mon[34885]: pgmap v440: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 24 MiB/s wr, 4.59k op/s 2024-09-21T05:50:54.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:53 smithi028 ceph-mon[31227]: pgmap v441: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 28 MiB/s wr, 5.61k op/s 2024-09-21T05:50:54.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:53 smithi181 ceph-mon[34819]: pgmap v441: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 28 MiB/s wr, 5.61k op/s 2024-09-21T05:50:54.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:53 smithi129 ceph-mon[34885]: pgmap v441: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 28 MiB/s wr, 5.61k op/s 2024-09-21T05:50:54.782 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:50:54.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:50:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 -- 172.21.15.28:0/2327430128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678077380 msgr2=0x7fd6780757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 --2- 172.21.15.28:0/2327430128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678077380 0x7fd6780757b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd674009900 tx=0x7fd67402a3f0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 -- 172.21.15.28:0/2327430128 shutdown_connections 2024-09-21T05:50:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 --2- 172.21.15.28:0/2327430128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678114430 0x7fd678116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 --2- 172.21.15.28:0/2327430128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6780761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 --2- 172.21.15.28:0/2327430128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678077380 0x7fd6780757b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.795+0000 7fd67f50e640 1 -- 172.21.15.28:0/2327430128 >> 172.21.15.28:0/2327430128 conn(0x7fd6780ff5d0 msgr2=0x7fd678101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.796+0000 7fd67f50e640 1 -- 172.21.15.28:0/2327430128 shutdown_connections 2024-09-21T05:50:55.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.796+0000 7fd67f50e640 1 -- 172.21.15.28:0/2327430128 wait complete. 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.797+0000 7fd67f50e640 1 Processor -- start 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.797+0000 7fd67f50e640 1 -- start start 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.797+0000 7fd67f50e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6781a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67f50e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 0x7fd6781a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67f50e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678114430 0x7fd6781a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67f50e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67811a0b0 con 0x7fd678077380 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67f50e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd678119f30 con 0x7fd678114430 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67f50e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd67811a230 con 0x7fd678075dd0 2024-09-21T05:50:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67d2aa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6781a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 0x7fd6781a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67d2aa640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6781a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:50812/0 (socket says 172.21.15.28:50812) 2024-09-21T05:50:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67d2aa640 1 -- 172.21.15.28:0/521101473 learned_addr learned my addr 172.21.15.28:0/521101473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.798+0000 7fd67daab640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678114430 0x7fd6781a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 -- 172.21.15.28:0/521101473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 msgr2=0x7fd6781a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6781a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 -- 172.21.15.28:0/521101473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678114430 msgr2=0x7fd6781a62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678114430 0x7fd6781a62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 -- 172.21.15.28:0/521101473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6781a69f0 con 0x7fd678077380 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.799+0000 7fd67caa9640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 0x7fd6781a1020 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd66800e9c0 tx=0x7fd66800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.800+0000 7fd66e7fc640 1 -- 172.21.15.28:0/521101473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66800ccc0 con 0x7fd678077380 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.800+0000 7fd66e7fc640 1 -- 172.21.15.28:0/521101473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6680108e0 con 0x7fd678077380 2024-09-21T05:50:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.800+0000 7fd66e7fc640 1 -- 172.21.15.28:0/521101473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680187c0 con 0x7fd678077380 2024-09-21T05:50:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.801+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6781a6ce0 con 0x7fd678077380 2024-09-21T05:50:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.801+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6781ad440 con 0x7fd678077380 2024-09-21T05:50:55.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.805+0000 7fd685a82740 1 Processor -- start 2024-09-21T05:50:55.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 -- start start 2024-09-21T05:50:55.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd66e7fc640 1 -- 172.21.15.28:0/521101473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd668010430 con 0x7fd678077380 2024-09-21T05:50:55.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642d5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642d5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f6424be60 con 0x560f63f0d670 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f6423b290 con 0x560f63f2e350 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.806+0000 7fd685a82740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560f6423f0a0 con 0x560f63f05490 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43232/0 (socket says 172.21.15.28:43232) 2024-09-21T05:50:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 -- 172.21.15.28:0/3685508695 learned_addr learned my addr 172.21.15.28:0/3685508695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 -- 172.21.15.28:0/3685508695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 msgr2=0x560f642d5970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642d5970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 -- 172.21.15.28:0/3685508695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 msgr2=0x560f642d5430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642d5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.807+0000 7fd67caa9640 1 -- 172.21.15.28:0/3685508695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f63f99930 con 0x560f63f0d670 2024-09-21T05:50:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.808+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd6680e5030 tx=0x7fd6680ef2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f233273e2fe42b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.808+0000 7fd6637fe640 1 -- 172.21.15.28:0/3685508695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680f6070 con 0x560f63f0d670 2024-09-21T05:50:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.808+0000 7fd6637fe640 1 -- 172.21.15.28:0/3685508695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6680f28d0 con 0x560f63f0d670 2024-09-21T05:50:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.808+0000 7fd6637fe640 1 -- 172.21.15.28:0/3685508695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680f1820 con 0x560f63f0d670 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.809+0000 7fd66e7fc640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 0x7fd6640b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.809+0000 7fd685a82740 1 -- 172.21.15.28:0/3685508695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 msgr2=0x560f642d5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.809+0000 7fd685a82740 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd6680e5030 tx=0x7fd6680ef2a0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.809+0000 7fd66e7fc640 1 -- 172.21.15.28:0/521101473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd6680e79e0 con 0x7fd678077380 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 -- 172.21.15.28:0/3685508695 shutdown_connections 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642d5eb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642d5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 --2- 172.21.15.28:0/3685508695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642d5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 -- 172.21.15.28:0/3685508695 >> 172.21.15.28:0/3685508695 conn(0x560f63f22740 msgr2=0x560f642d3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd67daab640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 0x7fd6640b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 -- 172.21.15.28:0/3685508695 shutdown_connections 2024-09-21T05:50:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 -- 172.21.15.28:0/3685508695 wait complete. 2024-09-21T05:50:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 -- start start 2024-09-21T05:50:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.810+0000 7fd685a82740 1 Processor -- start 2024-09-21T05:50:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd67daab640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 0x7fd6640b8cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd670005fd0 tx=0x7fd670009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ecf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f6409c890 con 0x560f63f05490 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f641d5d40 con 0x560f63f0d670 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560f642e3c10 con 0x560f63f2e350 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.811+0000 7fd685a82740 10 client.0 _ll_register_callbacks cb 0x560f64239c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67daab640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43240/0 (socket says 172.21.15.28:43240) 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 -- 172.21.15.28:0/3588993825 learned_addr learned my addr 172.21.15.28:0/3588993825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 -- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 msgr2=0x560f642ecf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ecf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 -- 172.21.15.28:0/3588993825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 msgr2=0x560f642e11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642e11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.812+0000 7fd67d2aa640 1 -- 172.21.15.28:0/3588993825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f642ed4c0 con 0x560f63f05490 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.813+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd67402a9e0 tx=0x7fd674004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57dbf25cf2c7a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.813+0000 7fd660ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.813+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674040070 con 0x560f63f05490 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.813+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd674005220 con 0x560f63f05490 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.813+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6680f28d0 con 0x560f63f05490 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.814+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd674035040 con 0x560f63f05490 2024-09-21T05:50:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.814+0000 7fd685a82740 10 client.16584 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.814+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560f642efcc0 con 0x560f63f05490 2024-09-21T05:50:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6680f28d0 con 0x560f63f05490 2024-09-21T05:50:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fd6740047a0 con 0x560f63f05490 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 10 client.16584 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 10 client.16584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 20 client.16584 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63eeccd0 0x560f642f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 4 client.16584 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f6424ddb0 con 0x560f63eeccd0 2024-09-21T05:50:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.815+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63eeccd0 0x560f642f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.817+0000 7fd660ff9640 10 client.16584 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.817+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63eeccd0 0x560f642f3750 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=112f6423b7f1700a server_cookie=a8df24bb004f0977 in_seq=0 out_seq=0 2024-09-21T05:50:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd660ff9640 1 -- 172.21.15.28:0/3588993825 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560f6424ddb0 con 0x560f63eeccd0 2024-09-21T05:50:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd660ff9640 10 client.16584 handle_command_reply: tid=0 2024-09-21T05:50:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 client.16584 shutdown 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 msgr2=0x560f642e0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd67402a9e0 tx=0x7fd674004340 comp rx=0 tx=0).stop 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 shutdown_connections 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63eeccd0 0x560f642f3750 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ecf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642e11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 --2- 172.21.15.28:0/3588993825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642e0c40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 >> 172.21.15.28:0/3588993825 conn(0x560f63f22740 msgr2=0x560f642d85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd660ff9640 0 client.16584 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.818+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 shutdown_connections 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.819+0000 7fd685a82740 1 -- 172.21.15.28:0/3588993825 wait complete. 2024-09-21T05:50:55.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.819+0000 7fd685a82740 20 client.16584 trim_cache size 0 max 16384 2024-09-21T05:50:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.849+0000 7fd685a82740 1 Processor -- start 2024-09-21T05:50:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.849+0000 7fd685a82740 1 -- start start 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ddef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b5cf0 con 0x560f63f05490 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b7980 con 0x560f63f0d670 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd685a82740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b63c0 con 0x560f63f2e350 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:41458/0 (socket says 172.21.15.28:41458) 2024-09-21T05:50:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.850+0000 7fd67caa9640 1 -- 172.21.15.28:0/3018126231 learned_addr learned my addr 172.21.15.28:0/3018126231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 -- 172.21.15.28:0/3018126231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 msgr2=0x560f642ddef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ddef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 -- 172.21.15.28:0/3018126231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 msgr2=0x560f642d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 -- 172.21.15.28:0/3018126231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f642ed2c0 con 0x560f63f0d670 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.851+0000 7fd67caa9640 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd6680f1490 tx=0x7fd6680f1420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e82d9b5a7625bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.852+0000 7fd662ffd640 1 -- 172.21.15.28:0/3018126231 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680f6070 con 0x560f63f0d670 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.852+0000 7fd662ffd640 1 -- 172.21.15.28:0/3018126231 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560f6424ddb0 con 0x560f63f0d670 2024-09-21T05:50:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.852+0000 7fd662ffd640 1 -- 172.21.15.28:0/3018126231 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680ed810 con 0x560f63f0d670 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.853+0000 7fd685a82740 1 -- 172.21.15.28:0/3018126231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 msgr2=0x560f642d8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.853+0000 7fd685a82740 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd6680f1490 tx=0x7fd6680f1420 comp rx=0 tx=0).stop 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 -- 172.21.15.28:0/3018126231 shutdown_connections 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f2e350 0x560f642ddef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f0d670 0x560f642d8c40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 --2- 172.21.15.28:0/3018126231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f05490 0x560f642d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 -- 172.21.15.28:0/3018126231 >> 172.21.15.28:0/3018126231 conn(0x560f63f22740 msgr2=0x560f642d8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.854+0000 7fd685a82740 1 -- 172.21.15.28:0/3018126231 shutdown_connections 2024-09-21T05:50:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.855+0000 7fd685a82740 1 -- 172.21.15.28:0/3018126231 wait complete. 2024-09-21T05:50:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.855+0000 7fd685a82740 1 -- start start 2024-09-21T05:50:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.855+0000 7fd685a82740 1 Processor -- start 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.856+0000 7fd685a82740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.856+0000 7fd685a82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.856+0000 7fd685a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642e58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd685a82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b71f0 con 0x560f63f0d670 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd67caa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43250/0 (socket says 172.21.15.28:43250) 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd67caa9640 1 -- 172.21.15.28:0/2480759601 learned_addr learned my addr 172.21.15.28:0/2480759601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b6af0 con 0x560f63f2e350 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781b3cd0 con 0x560f63f05490 2024-09-21T05:50:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.857+0000 7fd685a82740 10 client.0 _ll_register_callbacks cb 0x7fd668005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67daab640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642e58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67caa9640 1 -- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 msgr2=0x560f642e4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67caa9640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642e4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67caa9640 1 -- 172.21.15.28:0/2480759601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 msgr2=0x560f642e58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:55.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67caa9640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642e58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67caa9640 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6435d110 con 0x560f63f0d670 2024-09-21T05:50:55.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.858+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642e4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:50:55.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.859+0000 7fd67caa9640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd6680ad830 tx=0x7fd668010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a291cd941a5adcba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.859+0000 7fd660ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.859+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680f6030 con 0x560f63f0d670 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.859+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd6680a9b10 con 0x560f63f0d670 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.859+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560f6424ddb0 con 0x560f63f0d670 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.860+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd6680d2070 con 0x560f63f0d670 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.860+0000 7fd685a82740 10 client.16587 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:55.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.860+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560f6435e330 con 0x560f63f0d670 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.860+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd668010720 con 0x560f63f0d670 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fd6680efce0 con 0x560f63f0d670 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 10 client.16587 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 20 client.16587 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:55.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63ec2080 0x560f64360510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:55.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 4 client.16587 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:50:55.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560f6424ddb0 con 0x560f63ec2080 2024-09-21T05:50:55.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.861+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63ec2080 0x560f64360510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:55.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.863+0000 7fd660ff9640 10 client.16587 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:55.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.863+0000 7fd67d2aa640 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63ec2080 0x560f64360510 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843000d2464e05d6 server_cookie=35f9288699c3450a in_seq=0 out_seq=0 2024-09-21T05:50:55.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.865+0000 7fd660ff9640 1 -- 172.21.15.28:0/2480759601 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560f6424ddb0 con 0x560f63ec2080 2024-09-21T05:50:55.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.865+0000 7fd660ff9640 10 client.16587 handle_command_reply: tid=0 2024-09-21T05:50:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 client.16587 shutdown 2024-09-21T05:50:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 msgr2=0x560f642e5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd6680ad830 tx=0x7fd668010cc0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 shutdown_connections 2024-09-21T05:50:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x560f63ec2080 0x560f64360510 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f63f2e350 0x560f642e58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd660ff9640 0 client.16587 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f63f0d670 0x560f642e5360 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 --2- 172.21.15.28:0/2480759601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560f63f05490 0x560f642e4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 >> 172.21.15.28:0/2480759601 conn(0x560f63f22740 msgr2=0x560f642e20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 shutdown_connections 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 1 -- 172.21.15.28:0/2480759601 wait complete. 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.866+0000 7fd685a82740 20 client.16587 trim_cache size 0 max 16384 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:50:55.869 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:50:55.870 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "bacb5e14-d0ad-4b86-a228-c9c40a2eea63", 2024-09-21T05:50:55.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:50:55.870 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:50:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.867+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 msgr2=0x7fd6640b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.867+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 0x7fd6640b8cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd670005fd0 tx=0x7fd670009500 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.867+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 msgr2=0x7fd6781a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.867+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 0x7fd6781a1020 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd66800e9c0 tx=0x7fd66800ee80 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 shutdown_connections 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd678114430 0x7fd6781a62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6640b67b0 0x7fd6640b8cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678077380 0x7fd6781a1020 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 --2- 172.21.15.28:0/521101473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd678075dd0 0x7fd6781a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.868+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 >> 172.21.15.28:0/521101473 conn(0x7fd6780ff5d0 msgr2=0x7fd678110330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.869+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 shutdown_connections 2024-09-21T05:50:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:55.869+0000 7fd67f50e640 1 -- 172.21.15.28:0/521101473 wait complete. 2024-09-21T05:50:56.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:55 smithi181 ceph-mon[34819]: pgmap v442: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 16 MiB/s wr, 3.83k op/s 2024-09-21T05:50:56.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:55 smithi028 ceph-mon[31227]: pgmap v442: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 16 MiB/s wr, 3.83k op/s 2024-09-21T05:50:56.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:55 smithi129 ceph-mon[34885]: pgmap v442: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 16 MiB/s wr, 3.83k op/s 2024-09-21T05:50:56.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:50:57.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:50:57.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:56 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:56 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:57.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:56 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:58.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:57 smithi028 ceph-mon[31227]: pgmap v443: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 17 MiB/s wr, 4.05k op/s 2024-09-21T05:50:58.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:57 smithi181 ceph-mon[34819]: pgmap v443: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 17 MiB/s wr, 4.05k op/s 2024-09-21T05:50:58.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:57 smithi129 ceph-mon[34885]: pgmap v443: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 17 MiB/s wr, 4.05k op/s 2024-09-21T05:50:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.634+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4023796437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8102ae0 msgr2=0x7f16f8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.634+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4023796437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8102ae0 0x7f16f8102ef0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009900 tx=0x7f16ec02a410 comp rx=0 tx=0).stop 2024-09-21T05:50:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4023796437 shutdown_connections 2024-09-21T05:50:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4023796437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8108d20 0x7f16f810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4023796437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4023796437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8102ae0 0x7f16f8102ef0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4023796437 >> 172.21.15.28:0/4023796437 conn(0x7f16f8079790 msgr2=0x7f16f8079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4023796437 shutdown_connections 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.636+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4023796437 wait complete. 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.637+0000 7f16ffd6e640 1 Processor -- start 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.637+0000 7f16ffd6e640 1 -- start start 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8102ae0 0x7f16f810f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f810b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 0x7f16f810b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8111920 con 0x7f16f8108d20 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81117a0 con 0x7f16f81083d0 2024-09-21T05:50:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16ffd6e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8111aa0 con 0x7f16f8102ae0 2024-09-21T05:50:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16fd309640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f810b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 0x7f16f810b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 0x7f16f810b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43260/0 (socket says 172.21.15.28:43260) 2024-09-21T05:50:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16fe30b640 1 -- 172.21.15.28:0/4112330639 learned_addr learned my addr 172.21.15.28:0/4112330639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.638+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8102ae0 0x7f16f810f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 -- 172.21.15.28:0/4112330639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8102ae0 msgr2=0x7f16f810f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8102ae0 0x7f16f810f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 -- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 msgr2=0x7f16f810b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 0x7f16f810b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 -- 172.21.15.28:0/4112330639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f810be50 con 0x7f16f81083d0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16fd309640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f810b350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16f806ab20 tx=0x7f16f400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16eaffd640 1 -- 172.21.15.28:0/4112330639 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f400cd00 con 0x7f16f81083d0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16eaffd640 1 -- 172.21.15.28:0/4112330639 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16f40035f0 con 0x7f16f81083d0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.639+0000 7f16eaffd640 1 -- 172.21.15.28:0/4112330639 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4010690 con 0x7f16f81083d0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.640+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f810c140 con 0x7f16f81083d0 2024-09-21T05:50:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.640+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f81ad340 con 0x7f16f81083d0 2024-09-21T05:50:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.644+0000 7f17062e2740 1 Processor -- start 2024-09-21T05:50:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.644+0000 7f17062e2740 1 -- start start 2024-09-21T05:50:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f16eaffd640 1 -- 172.21.15.28:0/4112330639 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f16f40037a0 con 0x7f16f81083d0 2024-09-21T05:50:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f17062e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f31c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f31c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f31c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:41494/0 (socket says 172.21.15.28:41494) 2024-09-21T05:50:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f17062e2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f31c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.645+0000 7f16fe30b640 1 -- 172.21.15.28:0/3855910098 learned_addr learned my addr 172.21.15.28:0/3855910098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.646+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f31c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.646+0000 7f16eaffd640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 0x7f16e40b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.647+0000 7f16eaffd640 1 -- 172.21.15.28:0/4112330639 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f16f40e7930 con 0x7f16f81083d0 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.647+0000 7f17062e2740 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f31ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.647+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f292e50 con 0x55ae4ef54390 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.647+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f282280 con 0x55ae4ef75070 2024-09-21T05:50:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.647+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f286090 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fd309640 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f31ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fe30b640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 0x7f16e40b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 -- 172.21.15.28:0/3855910098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 msgr2=0x55ae4f31c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f31c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 -- 172.21.15.28:0/3855910098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 msgr2=0x55ae4f31ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f31ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 -- 172.21.15.28:0/3855910098 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae4efe0650 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.649+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f31c7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16f810edc0 tx=0x7f16ec032c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c3c73047882807 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.650+0000 7f16fe30b640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 0x7f16e40b8cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f16f0005c40 tx=0x7f16f0004ab0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.651+0000 7f16e3fff640 1 -- 172.21.15.28:0/3855910098 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec040070 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.651+0000 7f16e3fff640 1 -- 172.21.15.28:0/3855910098 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ec032e10 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.653+0000 7f16e3fff640 1 -- 172.21.15.28:0/3855910098 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec035040 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.653+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 msgr2=0x55ae4f31c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.653+0000 7f17062e2740 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f31c7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16f810edc0 tx=0x7f16ec032c20 comp rx=0 tx=0).stop 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 shutdown_connections 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f31ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f31c7b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 --2- 172.21.15.28:0/3855910098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f31c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 >> 172.21.15.28:0/3855910098 conn(0x55ae4ef69460 msgr2=0x55ae4f31a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 shutdown_connections 2024-09-21T05:50:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.654+0000 7f17062e2740 1 -- 172.21.15.28:0/3855910098 wait complete. 2024-09-21T05:50:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.655+0000 7f17062e2740 1 -- start start 2024-09-21T05:50:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.655+0000 7f17062e2740 1 Processor -- start 2024-09-21T05:50:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f338a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f344c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f0e3700 con 0x55ae4ef54390 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f21d110 con 0x55ae4ef75070 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ae4f33b950 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f17062e2740 10 client.0 _ll_register_callbacks cb 0x55ae4f236990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43278/0 (socket says 172.21.15.28:43278) 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 -- 172.21.15.28:0/1332067604 learned_addr learned my addr 172.21.15.28:0/1332067604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f338a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fe30b640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f344c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 -- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 msgr2=0x55ae4f338a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f338a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 -- 172.21.15.28:0/1332067604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 msgr2=0x55ae4f344c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f344c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.656+0000 7f16fd309640 1 -- 172.21.15.28:0/1332067604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae4f24ac40 con 0x55ae4ef54390 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f16fd309640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f16f4000c00 tx=0x7f16f40e55c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e8379ea199e92d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f16e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4009460 con 0x55ae4ef54390 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f16f40ac410 con 0x55ae4ef54390 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ec032e10 con 0x55ae4ef54390 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16f40ad250 con 0x55ae4ef54390 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f17062e2740 10 client.16599 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.657+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae4f3477a0 con 0x55ae4ef54390 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16f40ade40 con 0x55ae4ef54390 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f16f40aea70 con 0x55ae4ef54390 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 10 client.16599 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 20 client.16599 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef339f0 0x55ae4f34ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 4 client.16599 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f16ec032e10 con 0x55ae4ef339f0 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.658+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef339f0 0x55ae4f34ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.659+0000 7f16e17fa640 10 client.16599 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.659+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef339f0 0x55ae4f34ace0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e085a0dfc7f6978b server_cookie=99133657145ea2b5 in_seq=0 out_seq=0 2024-09-21T05:50:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.660+0000 7f16e17fa640 1 -- 172.21.15.28:0/1332067604 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f16ec032e10 con 0x55ae4ef339f0 2024-09-21T05:50:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.660+0000 7f16e17fa640 10 client.16599 handle_command_reply: tid=0 2024-09-21T05:50:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.660+0000 7f17062e2740 1 client.16599 shutdown 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.660+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 msgr2=0x55ae4f338f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.660+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f16f4000c00 tx=0x7f16f40e55c0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 shutdown_connections 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef339f0 0x55ae4f34ace0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef75070 0x55ae4f344c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f338f50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 --2- 172.21.15.28:0/1332067604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f338a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 >> 172.21.15.28:0/1332067604 conn(0x55ae4ef69460 msgr2=0x55ae4f31dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f16e17fa640 0 client.16599 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 shutdown_connections 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 1 -- 172.21.15.28:0/1332067604 wait complete. 2024-09-21T05:50:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.661+0000 7f17062e2740 20 client.16599 trim_cache size 0 max 16384 2024-09-21T05:50:58.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 Processor -- start 2024-09-21T05:50:58.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 -- start start 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 0x55ae4f335c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 0x55ae4f336140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81ad660 con 0x55ae4ef75070 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81af220 con 0x55ae4ef54390 2024-09-21T05:50:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f17062e2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81adc60 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f16fd309640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 0x55ae4f335c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f16fd309640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 0x55ae4f335c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:41514/0 (socket says 172.21.15.28:41514) 2024-09-21T05:50:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.686+0000 7f16fd309640 1 -- 172.21.15.28:0/2851770936 learned_addr learned my addr 172.21.15.28:0/2851770936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 0x55ae4f336140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 -- 172.21.15.28:0/2851770936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 msgr2=0x55ae4f33a640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 -- 172.21.15.28:0/2851770936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 msgr2=0x55ae4f335c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 0x55ae4f335c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.687+0000 7f16fe30b640 1 -- 172.21.15.28:0/2851770936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae4f31f030 con 0x55ae4ef75070 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.688+0000 7f16fe30b640 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 0x55ae4f336140 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f16f00049d0 tx=0x7f16f0009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0595fa635cb60a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.688+0000 7f16e37fe640 1 -- 172.21.15.28:0/2851770936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f0013170 con 0x55ae4ef75070 2024-09-21T05:50:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.688+0000 7f16e37fe640 1 -- 172.21.15.28:0/2851770936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16f000fcf0 con 0x55ae4ef75070 2024-09-21T05:50:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.688+0000 7f16e37fe640 1 -- 172.21.15.28:0/2851770936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f00056f0 con 0x55ae4ef75070 2024-09-21T05:50:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.689+0000 7f17062e2740 1 -- 172.21.15.28:0/2851770936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 msgr2=0x55ae4f336140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.689+0000 7f17062e2740 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 0x55ae4f336140 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f16f00049d0 tx=0x7f16f0009c10 comp rx=0 tx=0).stop 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 -- 172.21.15.28:0/2851770936 shutdown_connections 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef75070 0x55ae4f336140 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef54390 0x55ae4f335c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 --2- 172.21.15.28:0/2851770936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 -- 172.21.15.28:0/2851770936 >> 172.21.15.28:0/2851770936 conn(0x55ae4ef69460 msgr2=0x55ae4f339f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 -- 172.21.15.28:0/2851770936 shutdown_connections 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 -- 172.21.15.28:0/2851770936 wait complete. 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 -- start start 2024-09-21T05:50:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.690+0000 7f17062e2740 1 Processor -- start 2024-09-21T05:50:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f33f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 0x55ae4f33f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81aea90 con 0x55ae4ef54390 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16f81ae390 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16f80734f0 con 0x55ae4ef75070 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f17062e2740 10 client.0 _ll_register_callbacks cb 0x7f16f400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 0x55ae4f33f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f16fe30b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 0x55ae4f33f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:50886/0 (socket says 172.21.15.28:50886) 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f16fe30b640 1 -- 172.21.15.28:0/1312226203 learned_addr learned my addr 172.21.15.28:0/1312226203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f16fd309640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f33f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.691+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 -- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 msgr2=0x55ae4f33f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 0x55ae4f33f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 -- 172.21.15.28:0/1312226203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 msgr2=0x55ae4f33f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f33f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 -- 172.21.15.28:0/1312226203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae4f3b47d0 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16fdb0a640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33ecc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ae4f314560 tx=0x7f16ec032fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5eff61ae24fbc9e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.692+0000 7f16e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec035040 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f16ec0053c0 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16f000fcf0 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16ec040040 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f17062e2740 10 client.26192 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae4f3b5a40 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.693+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ec0055e0 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f16ec003c00 con 0x55ae4ef4c1b0 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 10 client.26192 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 20 client.26192 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef08da0 0x55ae4f3b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 4 client.26192 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:50:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f16f000fcf0 con 0x55ae4ef08da0 2024-09-21T05:50:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.694+0000 7f16fd309640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef08da0 0x55ae4f3b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.695+0000 7f16fd309640 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef08da0 0x55ae4f3b7e10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec1c054816d12323 server_cookie=a82971a35123fbcc in_seq=0 out_seq=0 2024-09-21T05:50:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.695+0000 7f16e17fa640 10 client.26192 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.696+0000 7f16e17fa640 1 -- 172.21.15.28:0/1312226203 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2281 (crc 0 0 0) 0x7f16f000fcf0 con 0x55ae4ef08da0 2024-09-21T05:50:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.696+0000 7f16e17fa640 10 client.26192 handle_command_reply: tid=0 2024-09-21T05:50:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.697+0000 7f17062e2740 1 client.26192 shutdown 2024-09-21T05:50:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.697+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 msgr2=0x55ae4f33ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.697+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33ecc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55ae4f314560 tx=0x7f16ec032fc0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 shutdown_connections 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ae4ef08da0 0x55ae4f3b7e10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ae4ef75070 0x55ae4f33f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f16e17fa640 0 client.26192 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4ef54390 0x55ae4f33f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 --2- 172.21.15.28:0/1312226203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae4ef4c1b0 0x55ae4f33ecc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.698+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 >> 172.21.15.28:0/1312226203 conn(0x55ae4ef69460 msgr2=0x55ae4f339db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.699+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 shutdown_connections 2024-09-21T05:50:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.700+0000 7f17062e2740 1 -- 172.21.15.28:0/1312226203 wait complete. 2024-09-21T05:50:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.700+0000 7f17062e2740 20 client.26192 trim_cache size 0 max 16384 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (60 inodes in the stack)", 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "bacb5e14-d0ad-4b86-a228-c9c40a2eea63", 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.704 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:50:58.706 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.707 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:50:58.708 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 msgr2=0x7f16e40b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 0x7f16e40b8cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f16f0005c40 tx=0x7f16f0004ab0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 msgr2=0x7f16f810b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f810b350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16f806ab20 tx=0x7f16f400ee50 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 shutdown_connections 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f16e40b67b0 0x7f16e40b8cc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f8108d20 0x7f16f810b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f81083d0 0x7f16f810b350 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 --2- 172.21.15.28:0/4112330639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f8102ae0 0x7f16f810f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 >> 172.21.15.28:0/4112330639 conn(0x7f16f8079790 msgr2=0x7f16f81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 shutdown_connections 2024-09-21T05:50:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:50:58.705+0000 7f16ffd6e640 1 -- 172.21.15.28:0/4112330639 wait complete. 2024-09-21T05:50:59.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:58 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:59.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:58 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:59.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:58 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 -- 172.21.15.28:0/3961605090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14106b10 msgr2=0x7fbc14110580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 --2- 172.21.15.28:0/3961605090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14106b10 0x7fbc14110580 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009ca0 tx=0x7fbc0c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 -- 172.21.15.28:0/3961605090 shutdown_connections 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 --2- 172.21.15.28:0/3961605090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14106b10 0x7fbc14110580 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 --2- 172.21.15.28:0/3961605090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc141061c0 0x7fbc141065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 --2- 172.21.15.28:0/3961605090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc1406c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.763+0000 7fbc1a720640 1 -- 172.21.15.28:0/3961605090 >> 172.21.15.28:0/3961605090 conn(0x7fbc140784e0 msgr2=0x7fbc140788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.764+0000 7fbc1a720640 1 -- 172.21.15.28:0/3961605090 shutdown_connections 2024-09-21T05:50:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.764+0000 7fbc1a720640 1 -- 172.21.15.28:0/3961605090 wait complete. 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.764+0000 7fbc1a720640 1 Processor -- start 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 -- start start 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc141061c0 0x7fbc141027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc14106b10 0x7fbc14102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113a10 con 0x7fbc141061c0 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113890 con 0x7fbc14106b10 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc1a720640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113b90 con 0x7fbc1406bd10 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.765+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.766+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:50902/0 (socket says 172.21.15.28:50902) 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.766+0000 7fbc13fff640 1 -- 172.21.15.28:0/1973508054 learned_addr learned my addr 172.21.15.28:0/1973508054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.766+0000 7fbc137fe640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc141061c0 0x7fbc141027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc14106b10 0x7fbc14102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 -- 172.21.15.28:0/1973508054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc14106b10 msgr2=0x7fbc14102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc14106b10 0x7fbc14102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 -- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc141061c0 msgr2=0x7fbc141027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc141061c0 0x7fbc141027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 -- 172.21.15.28:0/1973508054 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc14103510 con 0x7fbc1406bd10 2024-09-21T05:50:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.767+0000 7fbc13fff640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d950 tx=0x7fbc0400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.768+0000 7fbc117fa640 1 -- 172.21.15.28:0/1973508054 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04017070 con 0x7fbc1406bd10 2024-09-21T05:50:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.768+0000 7fbc117fa640 1 -- 172.21.15.28:0/1973508054 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc04009c30 con 0x7fbc1406bd10 2024-09-21T05:50:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.768+0000 7fbc117fa640 1 -- 172.21.15.28:0/1973508054 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0400f620 con 0x7fbc1406bd10 2024-09-21T05:50:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.768+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc141acd20 con 0x7fbc1406bd10 2024-09-21T05:50:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.768+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc141ad1c0 con 0x7fbc1406bd10 2024-09-21T05:50:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 Processor -- start 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 -- start start 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94ddd4d0 0x562c951a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c951186f0 con 0x562c94dd52f0 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c95107b20 con 0x562c94dfe1b0 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc2168c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c9510b930 con 0x562c94ddd4d0 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc18cbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc18cbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:41538/0 (socket says 172.21.15.28:41538) 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc18cbd640 1 -- 172.21.15.28:0/1200435848 learned_addr learned my addr 172.21.15.28:0/1200435848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.772+0000 7fbc137fe640 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94ddd4d0 0x562c951a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 -- 172.21.15.28:0/1200435848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94ddd4d0 msgr2=0x562c951a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94ddd4d0 0x562c951a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 -- 172.21.15.28:0/1200435848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 msgr2=0x562c951a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 -- 172.21.15.28:0/1200435848 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c950cd530 con 0x562c94dfe1b0 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.773+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006010 tx=0x7fbc0c009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0dceac9cc045110 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.774+0000 7fbc117fa640 1 -- 172.21.15.28:0/1973508054 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbc04010460 con 0x7fbc1406bd10 2024-09-21T05:50:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.774+0000 7fbbfa7fc640 1 -- 172.21.15.28:0/1200435848 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c031cf0 con 0x562c94dfe1b0 2024-09-21T05:50:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.774+0000 7fbbfa7fc640 1 -- 172.21.15.28:0/1200435848 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc0c03f070 con 0x562c94dfe1b0 2024-09-21T05:50:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.774+0000 7fbbfa7fc640 1 -- 172.21.15.28:0/1200435848 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0042a0 con 0x562c94dfe1b0 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.775+0000 7fbc2168c740 1 -- 172.21.15.28:0/1200435848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 msgr2=0x562c951a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.775+0000 7fbc2168c740 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006010 tx=0x7fbc0c009800 comp rx=0 tx=0).stop 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.775+0000 7fbc2168c740 1 -- 172.21.15.28:0/1200435848 shutdown_connections 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94ddd4d0 0x562c951a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 --2- 172.21.15.28:0/1200435848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dfe1b0 0x562c951a39b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 -- 172.21.15.28:0/1200435848 >> 172.21.15.28:0/1200435848 conn(0x562c94df2300 msgr2=0x562c951a18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc117fa640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 0x7fbbfc0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 -- 172.21.15.28:0/1200435848 shutdown_connections 2024-09-21T05:50:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.776+0000 7fbc2168c740 1 -- 172.21.15.28:0/1200435848 wait complete. 2024-09-21T05:50:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.777+0000 7fbc117fa640 1 -- 172.21.15.28:0/1973508054 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc040e72c0 con 0x7fbc1406bd10 2024-09-21T05:50:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.777+0000 7fbc137fe640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 0x7fbbfc0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.777+0000 7fbc2168c740 1 -- start start 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.777+0000 7fbc2168c740 1 Processor -- start 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c9503fab0 con 0x562c94ddd4d0 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c950ad730 con 0x562c94dd52f0 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562c951c30e0 con 0x562c94dfe1b0 2024-09-21T05:50:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc2168c740 10 client.0 _ll_register_callbacks cb 0x562c951016b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:41550/0 (socket says 172.21.15.28:41550) 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc13fff640 1 -- 172.21.15.28:0/2510142099 learned_addr learned my addr 172.21.15.28:0/2510142099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.778+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc13fff640 1 -- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 msgr2=0x562c951cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc13fff640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc13fff640 1 -- 172.21.15.28:0/2510142099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 msgr2=0x562c951af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc13fff640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc13fff640 1 -- 172.21.15.28:0/2510142099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c94e69790 con 0x562c94dd52f0 2024-09-21T05:50:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.779+0000 7fbc137fe640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 0x7fbbfc0b8d60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406aec0 tx=0x7fbc0800bbf0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbc13fff640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc04000c00 tx=0x7fbc040e4f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fdc8a5c108db693 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbbf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04020040 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc0c03f070 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc04010750 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbc040acbe0 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbc2168c740 10 client.26201 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.780+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562c94ee2620 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.781+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc0c03f070 con 0x562c94dd52f0 2024-09-21T05:50:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.781+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fbc04010950 con 0x562c94dd52f0 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 10 client.26201 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 10 client.26201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 20 client.26201 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94dbcd80 0x562c951d2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 4 client.26201 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.782+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c9511a640 con 0x562c94dbcd80 2024-09-21T05:50:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.783+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94dbcd80 0x562c951d2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.785+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94dbcd80 0x562c951d2370 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5bc8bef6441dad server_cookie=3bd86d218c6870de in_seq=0 out_seq=0 2024-09-21T05:50:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.786+0000 7fbbf37fe640 10 client.26201 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.787+0000 7fbbf37fe640 1 -- 172.21.15.28:0/2510142099 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562c9511a640 con 0x562c94dbcd80 2024-09-21T05:50:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.787+0000 7fbbf37fe640 10 client.26201 handle_command_reply: tid=0 2024-09-21T05:50:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.788+0000 7fbc2168c740 1 client.26201 shutdown 2024-09-21T05:50:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.790+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 msgr2=0x562c951af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.790+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc04000c00 tx=0x7fbc040e4f50 comp rx=0 tx=0).stop 2024-09-21T05:50:59.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.790+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 shutdown_connections 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.790+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94dbcd80 0x562c951d2370 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.790+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 1 --2- 172.21.15.28:0/2510142099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951af180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 >> 172.21.15.28:0/2510142099 conn(0x562c94df2300 msgr2=0x562c951a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbbf37fe640 0 client.26201 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 shutdown_connections 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 1 -- 172.21.15.28:0/2510142099 wait complete. 2024-09-21T05:50:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.793+0000 7fbc2168c740 20 client.26201 trim_cache size 0 max 16384 2024-09-21T05:50:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.824+0000 7fbc2168c740 1 Processor -- start 2024-09-21T05:50:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.824+0000 7fbc2168c740 1 -- start start 2024-09-21T05:50:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.824+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.824+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc2168c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141ad3f0 con 0x562c94ddd4d0 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc2168c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141af060 con 0x562c94dd52f0 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc2168c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141adaa0 con 0x562c94dfe1b0 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43342/0 (socket says 172.21.15.28:43342) 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 -- 172.21.15.28:0/436789653 learned_addr learned my addr 172.21.15.28:0/436789653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc18cbd640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc13fff640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 -- 172.21.15.28:0/436789653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 msgr2=0x562c951cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 -- 172.21.15.28:0/436789653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 msgr2=0x562c951b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc137fe640 1 -- 172.21.15.28:0/436789653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c94dbcfb0 con 0x562c94ddd4d0 2024-09-21T05:50:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.825+0000 7fbc13fff640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.826+0000 7fbc137fe640 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406b790 tx=0x7fbc0801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1984f5a7ea245664 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.826+0000 7fbbf9ffb640 1 -- 172.21.15.28:0/436789653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08021e00 con 0x562c94ddd4d0 2024-09-21T05:50:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.826+0000 7fbbf9ffb640 1 -- 172.21.15.28:0/436789653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc08021110 con 0x562c94ddd4d0 2024-09-21T05:50:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.826+0000 7fbbf9ffb640 1 -- 172.21.15.28:0/436789653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0802a730 con 0x562c94ddd4d0 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.827+0000 7fbc2168c740 1 -- 172.21.15.28:0/436789653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 msgr2=0x562c951cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.827+0000 7fbc2168c740 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406b790 tx=0x7fbc0801e000 comp rx=0 tx=0).stop 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.827+0000 7fbc2168c740 1 -- 172.21.15.28:0/436789653 shutdown_connections 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c951cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94ddd4d0 0x562c951cc2d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 --2- 172.21.15.28:0/436789653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94dd52f0 0x562c951b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 -- 172.21.15.28:0/436789653 >> 172.21.15.28:0/436789653 conn(0x562c94df2300 msgr2=0x562c951b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 -- 172.21.15.28:0/436789653 shutdown_connections 2024-09-21T05:50:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.828+0000 7fbc2168c740 1 -- 172.21.15.28:0/436789653 wait complete. 2024-09-21T05:50:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.829+0000 7fbc2168c740 1 -- start start 2024-09-21T05:50:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.829+0000 7fbc2168c740 1 Processor -- start 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 0x562c951c6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c95233cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141ae8d0 con 0x562c94dd52f0 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141ae1d0 con 0x562c94ddd4d0 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141a4c50 con 0x562c94dfe1b0 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc2168c740 10 client.0 _ll_register_callbacks cb 0x7fbc0400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:50:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc18cbd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c95233cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc18cbd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c95233cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:50940/0 (socket says 172.21.15.28:50940) 2024-09-21T05:50:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc18cbd640 1 -- 172.21.15.28:0/4097500271 learned_addr learned my addr 172.21.15.28:0/4097500271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:50:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.830+0000 7fbc137fe640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 0x562c951c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 -- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 msgr2=0x562c95233cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c95233cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 -- 172.21.15.28:0/4097500271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 msgr2=0x562c951c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 -- 172.21.15.28:0/4097500271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c95234340 con 0x562c94ddd4d0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc13fff640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbc137fe640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 0x562c951c6390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406b790 tx=0x7fbc08005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9adc379775b7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbbf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc080212c0 con 0x562c94ddd4d0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.831+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fbc0802d040 con 0x562c94ddd4d0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.832+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc08021110 con 0x562c94ddd4d0 2024-09-21T05:50:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.832+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbc0801f450 con 0x562c94ddd4d0 2024-09-21T05:50:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.832+0000 7fbc2168c740 10 client.26207 fetch_fsmap learned FSMap version 36 2024-09-21T05:50:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.832+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562c9523c870 con 0x562c94ddd4d0 2024-09-21T05:50:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.832+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc08021110 con 0x562c94ddd4d0 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7fbc0802ab10 con 0x562c94ddd4d0 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 10 client.26207 fetch_fsmap finished waiting for FSMap version 36 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 10 client.26207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 20 client.26207 populate_metadata read hostname 'smithi028' 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94d921c0 0x562c9523ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 4 client.26207 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c951160f0 con 0x562c94d921c0 2024-09-21T05:50:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.833+0000 7fbc13fff640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94d921c0 0x562c9523ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:50:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.836+0000 7fbc13fff640 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94d921c0 0x562c9523ef40 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a4f6c123f43117d server_cookie=4a962548647f05c in_seq=0 out_seq=0 2024-09-21T05:50:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.836+0000 7fbbf37fe640 10 client.26207 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.937+0000 7fbbf37fe640 1 -- 172.21.15.28:0/4097500271 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c951160f0 con 0x562c94d921c0 2024-09-21T05:50:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.937+0000 7fbbf37fe640 10 client.26207 handle_command_reply: tid=0 2024-09-21T05:50:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.937+0000 7fbc2168c740 1 client.26207 shutdown 2024-09-21T05:50:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.937+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 msgr2=0x562c951c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 0x562c951c6390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406b790 tx=0x7fbc08005740 comp rx=0 tx=0).stop 2024-09-21T05:50:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 shutdown_connections 2024-09-21T05:50:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562c94d921c0 0x562c9523ef40 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbbf37fe640 0 client.26207 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562c94dfe1b0 0x562c95233cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c94ddd4d0 0x562c951c6390 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 --2- 172.21.15.28:0/4097500271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c94dd52f0 0x562c951c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 >> 172.21.15.28:0/4097500271 conn(0x562c94df2300 msgr2=0x562c951cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 shutdown_connections 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 1 -- 172.21.15.28:0/4097500271 wait complete. 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.938+0000 7fbc2168c740 20 client.26207 trim_cache size 0 max 16384 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:50:59.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:50:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:50:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 msgr2=0x7fbbfc0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 0x7fbbfc0b8d60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbc1406aec0 tx=0x7fbc0800bbf0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 msgr2=0x7fbc141041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d950 tx=0x7fbc0400de10 comp rx=0 tx=0).stop 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 shutdown_connections 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc14106b10 0x7fbc14102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbbfc0b6850 0x7fbbfc0b8d60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc141061c0 0x7fbc141027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 --2- 172.21.15.28:0/1973508054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1406bd10 0x7fbc141041d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 >> 172.21.15.28:0/1973508054 conn(0x7fbc140784e0 msgr2=0x7fbc141073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.939+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 shutdown_connections 2024-09-21T05:50:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:50:59.940+0000 7fbc1a720640 1 -- 172.21.15.28:0/1973508054 wait complete. 2024-09-21T05:51:00.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:50:59 smithi181 ceph-mon[34819]: pgmap v444: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 22 MiB/s wr, 5.23k op/s 2024-09-21T05:51:00.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:50:59 smithi028 ceph-mon[31227]: pgmap v444: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 22 MiB/s wr, 5.23k op/s 2024-09-21T05:51:00.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:50:59 smithi129 ceph-mon[34885]: pgmap v444: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 22 MiB/s wr, 5.23k op/s 2024-09-21T05:51:01.120 INFO:tasks.cephfs.filesystem:scrub status for tag:bacb5e14-d0ad-4b86-a228-c9c40a2eea63 - {'path': '/', 'tag': 'bacb5e14-d0ad-4b86-a228-c9c40a2eea63', 'options': 'recursive,force'} 2024-09-21T05:51:01.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:00 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:01.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:00 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:01.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:00 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:02.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:01 smithi181 ceph-mon[34819]: pgmap v445: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 21 MiB/s wr, 4.90k op/s 2024-09-21T05:51:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:01 smithi028 ceph-mon[31227]: pgmap v445: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 21 MiB/s wr, 4.90k op/s 2024-09-21T05:51:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:01 smithi129 ceph-mon[34885]: pgmap v445: 97 pgs: 97 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 21 MiB/s wr, 4.90k op/s 2024-09-21T05:51:04.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:03 smithi181 ceph-mon[34819]: pgmap v446: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 38 MiB/s wr, 8.62k op/s 2024-09-21T05:51:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:03 smithi028 ceph-mon[31227]: pgmap v446: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 38 MiB/s wr, 8.62k op/s 2024-09-21T05:51:04.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:03 smithi129 ceph-mon[34885]: pgmap v446: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 38 MiB/s wr, 8.62k op/s 2024-09-21T05:51:06.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:05 smithi181 ceph-mon[34819]: pgmap v447: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 37 MiB/s wr, 8.35k op/s 2024-09-21T05:51:06.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:05 smithi028 ceph-mon[31227]: pgmap v447: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 37 MiB/s wr, 8.35k op/s 2024-09-21T05:51:06.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:05 smithi129 ceph-mon[34885]: pgmap v447: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 37 MiB/s wr, 8.35k op/s 2024-09-21T05:51:07.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:51:07.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:51:07.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:51:07.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:51:07.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:51:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:51:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:51:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:51:07.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:51:07.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:51:07.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:51:07.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:51:07.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:51:07.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:51:07.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:51:08.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:08 smithi181 ceph-mon[34819]: pgmap v448: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 37 MiB/s wr, 8.24k op/s 2024-09-21T05:51:08.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:08 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:08 smithi028 ceph-mon[31227]: pgmap v448: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 37 MiB/s wr, 8.24k op/s 2024-09-21T05:51:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:08 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:08.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:08 smithi129 ceph-mon[34885]: pgmap v448: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 37 MiB/s wr, 8.24k op/s 2024-09-21T05:51:08.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:08 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:10.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:10 smithi181 ceph-mon[34819]: pgmap v449: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 47 MiB/s wr, 10.43k op/s 2024-09-21T05:51:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:10 smithi028 ceph-mon[31227]: pgmap v449: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 47 MiB/s wr, 10.43k op/s 2024-09-21T05:51:10.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:10 smithi129 ceph-mon[34885]: pgmap v449: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 47 MiB/s wr, 10.43k op/s 2024-09-21T05:51:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:12 smithi181 ceph-mon[34819]: pgmap v450: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-09-21T05:51:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:12 smithi028 ceph-mon[31227]: pgmap v450: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-09-21T05:51:12.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:12 smithi129 ceph-mon[34885]: pgmap v450: 97 pgs: 97 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.126+0000 7f44ca790640 1 -- 172.21.15.28:0/1180737757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 msgr2=0x7f44c4113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.126+0000 7f44ca790640 1 --2- 172.21.15.28:0/1180737757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c4113fd0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009900 tx=0x7f44ac02a440 comp rx=0 tx=0).stop 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.127+0000 7f44ca790640 1 -- 172.21.15.28:0/1180737757 shutdown_connections 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.127+0000 7f44ca790640 1 --2- 172.21.15.28:0/1180737757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c4116930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.127+0000 7f44ca790640 1 --2- 172.21.15.28:0/1180737757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c4113fd0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.127+0000 7f44ca790640 1 --2- 172.21.15.28:0/1180737757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 0x7f44c4101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.127+0000 7f44ca790640 1 -- 172.21.15.28:0/1180737757 >> 172.21.15.28:0/1180737757 conn(0x7f44c4077770 msgr2=0x7f44c4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.129+0000 7f44ca790640 1 -- 172.21.15.28:0/1180737757 shutdown_connections 2024-09-21T05:51:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.129+0000 7f44ca790640 1 -- 172.21.15.28:0/1180737757 wait complete. 2024-09-21T05:51:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.130+0000 7f44ca790640 1 Processor -- start 2024-09-21T05:51:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.130+0000 7f44ca790640 1 -- start start 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 0x7f44c4074fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c406f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44c406d6e0 con 0x7f44c4102140 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44c406d560 con 0x7f44c4114510 2024-09-21T05:51:13.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44ca790640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44c406d860 con 0x7f44c41017f0 2024-09-21T05:51:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.131+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42520/0 (socket says 172.21.15.28:42520) 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 -- 172.21.15.28:0/2183107383 learned_addr learned my addr 172.21.15.28:0/2183107383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c3fff640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 0x7f44c4074fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c37fe640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 -- 172.21.15.28:0/2183107383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 msgr2=0x7f44c4074fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 0x7f44c4074fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 -- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 msgr2=0x7f44c406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c406f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c8d2d640 1 -- 172.21.15.28:0/2183107383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c41b5520 con 0x7f44c4114510 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.132+0000 7f44c37fe640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c406f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:51:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b970 tx=0x7f44bc00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44c17fa640 1 -- 172.21.15.28:0/2183107383 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc019070 con 0x7f44c4114510 2024-09-21T05:51:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44c17fa640 1 -- 172.21.15.28:0/2183107383 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44bc014440 con 0x7f44c4114510 2024-09-21T05:51:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44c17fa640 1 -- 172.21.15.28:0/2183107383 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0075f0 con 0x7f44c4114510 2024-09-21T05:51:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c41b56f0 con 0x7f44c4114510 2024-09-21T05:51:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.133+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c41b5c40 con 0x7f44c4114510 2024-09-21T05:51:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.138+0000 7f44c17fa640 1 -- 172.21.15.28:0/2183107383 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f44bc01c070 con 0x7f44c4114510 2024-09-21T05:51:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.139+0000 7f44c17fa640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 0x7f44b00b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.139+0000 7f44c17fa640 1 -- 172.21.15.28:0/2183107383 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44bc0e6a70 con 0x7f44c4114510 2024-09-21T05:51:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.142+0000 7f44d17dd740 1 Processor -- start 2024-09-21T05:51:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.142+0000 7f44d17dd740 1 -- start start 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.142+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a58c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.142+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.142+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44d17dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b2a5016f0 con 0x562b2a1be2f0 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44d17dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562b2a4f0b20 con 0x562b2a1c64d0 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44d17dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562b2a4f4930 con 0x562b2a1e71b0 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42536/0 (socket says 172.21.15.28:42536) 2024-09-21T05:51:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.143+0000 7f44c8d2d640 1 -- 172.21.15.28:0/4151701618 learned_addr learned my addr 172.21.15.28:0/4151701618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 -- 172.21.15.28:0/4151701618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 msgr2=0x562b2a58c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a58c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 -- 172.21.15.28:0/4151701618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 msgr2=0x562b2a58cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 -- 172.21.15.28:0/4151701618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2a4b6530 con 0x562b2a1c64d0 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.144+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0ac8c0 tx=0x7f44bc0ac280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3821bb0e7c793a91 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.145+0000 7f44c3fff640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 0x7f44b00b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.147+0000 7f44ba7fc640 1 -- 172.21.15.28:0/4151701618 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0ac440 con 0x562b2a1c64d0 2024-09-21T05:51:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.147+0000 7f44ba7fc640 1 -- 172.21.15.28:0/4151701618 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44bc0b1360 con 0x562b2a1c64d0 2024-09-21T05:51:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.147+0000 7f44c3fff640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 0x7f44b00b8cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009730 tx=0x7f44b40096c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.147+0000 7f44ba7fc640 1 -- 172.21.15.28:0/4151701618 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0b6070 con 0x562b2a1c64d0 2024-09-21T05:51:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.148+0000 7f44d17dd740 1 -- 172.21.15.28:0/4151701618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 msgr2=0x562b2a58d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.149+0000 7f44d17dd740 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0ac8c0 tx=0x7f44bc0ac280 comp rx=0 tx=0).stop 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.149+0000 7f44d17dd740 1 -- 172.21.15.28:0/4151701618 shutdown_connections 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.150+0000 7f44d17dd740 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a58d430 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.150+0000 7f44d17dd740 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.150+0000 7f44d17dd740 1 --2- 172.21.15.28:0/4151701618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a58c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.150+0000 7f44d17dd740 1 -- 172.21.15.28:0/4151701618 >> 172.21.15.28:0/4151701618 conn(0x562b2a1db300 msgr2=0x562b2a58a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.150+0000 7f44d17dd740 1 -- 172.21.15.28:0/4151701618 shutdown_connections 2024-09-21T05:51:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.151+0000 7f44d17dd740 1 -- 172.21.15.28:0/4151701618 wait complete. 2024-09-21T05:51:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.151+0000 7f44d17dd740 1 -- start start 2024-09-21T05:51:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.152+0000 7f44d17dd740 1 Processor -- start 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 0x562b2a598020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a598560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b2a428ab0 con 0x562b2a1c64d0 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562b2a496730 con 0x562b2a1e71b0 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562b2a59b180 con 0x562b2a1be2f0 2024-09-21T05:51:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.153+0000 7f44d17dd740 10 client.0 _ll_register_callbacks cb 0x562b2a4ea6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42538/0 (socket says 172.21.15.28:42538) 2024-09-21T05:51:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c8d2d640 1 -- 172.21.15.28:0/1585659635 learned_addr learned my addr 172.21.15.28:0/1585659635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 0x562b2a598020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c37fe640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a598560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 -- 172.21.15.28:0/1585659635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 msgr2=0x562b2a5a4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 -- 172.21.15.28:0/1585659635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 msgr2=0x562b2a598560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a598560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.154+0000 7f44c3fff640 1 -- 172.21.15.28:0/1585659635 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2a252790 con 0x562b2a1be2f0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.155+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 0x562b2a598020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44c4074ce0 tx=0x7f44b4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66cd68b8d3faf11 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.155+0000 7f44a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.155+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b401e9b0 con 0x562b2a1be2f0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.155+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44b401eb50 con 0x562b2a1be2f0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.155+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44bc0b1360 con 0x562b2a1be2f0 2024-09-21T05:51:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.156+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f44b401ed50 con 0x562b2a1be2f0 2024-09-21T05:51:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.158+0000 7f44d17dd740 10 client.26224 fetch_fsmap learned FSMap version 38 2024-09-21T05:51:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562b2a2cb620 con 0x562b2a1be2f0 2024-09-21T05:51:13.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44bc0b1360 con 0x562b2a1be2f0 2024-09-21T05:51:13.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f44b401d3b0 con 0x562b2a1be2f0 2024-09-21T05:51:13.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 10 client.26224 fetch_fsmap finished waiting for FSMap version 38 2024-09-21T05:51:13.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 10 client.26224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:13.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 20 client.26224 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a1a5d80 0x562b2a5aa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 4 client.26224 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.159+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b2a503640 con 0x562b2a1a5d80 2024-09-21T05:51:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.160+0000 7f44c37fe640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a1a5d80 0x562b2a5aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.161+0000 7f44a37fe640 10 client.26224 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.163+0000 7f44c37fe640 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a1a5d80 0x562b2a5aa380 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b75f1336a85c344 server_cookie=2fee3041bd8c2be5 in_seq=0 out_seq=0 2024-09-21T05:51:13.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.163+0000 7f44a37fe640 1 -- 172.21.15.28:0/1585659635 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562b2a503640 con 0x562b2a1a5d80 2024-09-21T05:51:13.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44a37fe640 10 client.26224 handle_command_reply: tid=0 2024-09-21T05:51:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 client.26224 shutdown 2024-09-21T05:51:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 msgr2=0x562b2a598020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 0x562b2a598020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44c4074ce0 tx=0x7f44b4005f90 comp rx=0 tx=0).stop 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 shutdown_connections 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a1a5d80 0x562b2a5aa380 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a598560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1585659635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1be2f0 0x562b2a598020 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 >> 172.21.15.28:0/1585659635 conn(0x562b2a1db300 msgr2=0x562b2a58d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.164+0000 7f44a37fe640 0 client.26224 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.165+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 shutdown_connections 2024-09-21T05:51:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.165+0000 7f44d17dd740 1 -- 172.21.15.28:0/1585659635 wait complete. 2024-09-21T05:51:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.165+0000 7f44d17dd740 20 client.26224 trim_cache size 0 max 16384 2024-09-21T05:51:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.198+0000 7f44d17dd740 1 Processor -- start 2024-09-21T05:51:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.198+0000 7f44d17dd740 1 -- start start 2024-09-21T05:51:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44d17dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44a400e7a0 con 0x562b2a1be2f0 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44c37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44c37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42554/0 (socket says 172.21.15.28:42554) 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44c37fe640 1 -- 172.21.15.28:0/2424547308 learned_addr learned my addr 172.21.15.28:0/2424547308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.199+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.200+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44a40103a0 con 0x562b2a1c64d0 2024-09-21T05:51:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.200+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44a400ee70 con 0x562b2a1e71b0 2024-09-21T05:51:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.201+0000 7f44c37fe640 1 -- 172.21.15.28:0/2424547308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 msgr2=0x562b2a5a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.201+0000 7f44c37fe640 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.201+0000 7f44c37fe640 1 -- 172.21.15.28:0/2424547308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 msgr2=0x562b2a58e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.201+0000 7f44c37fe640 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.201+0000 7f44c37fe640 1 -- 172.21.15.28:0/2424547308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2a596500 con 0x562b2a1c64d0 2024-09-21T05:51:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.202+0000 7f44c37fe640 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f44ac02aa30 tx=0x7f44ac000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=524a4694cb980114 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.203+0000 7f44b9ffb640 1 -- 172.21.15.28:0/2424547308 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac031640 con 0x562b2a1c64d0 2024-09-21T05:51:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.203+0000 7f44b9ffb640 1 -- 172.21.15.28:0/2424547308 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44ac031c90 con 0x562b2a1c64d0 2024-09-21T05:51:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.203+0000 7f44b9ffb640 1 -- 172.21.15.28:0/2424547308 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac03db70 con 0x562b2a1c64d0 2024-09-21T05:51:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.206+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 msgr2=0x562b2a5a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.206+0000 7f44d17dd740 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f44ac02aa30 tx=0x7f44ac000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.210+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 shutdown_connections 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.210+0000 7f44d17dd740 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a5a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.210+0000 7f44d17dd740 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1c64d0 0x562b2a5a4420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.210+0000 7f44d17dd740 1 --2- 172.21.15.28:0/2424547308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1be2f0 0x562b2a58e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.210+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 >> 172.21.15.28:0/2424547308 conn(0x562b2a1db300 msgr2=0x562b2a58d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.211+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 shutdown_connections 2024-09-21T05:51:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.213+0000 7f44d17dd740 1 -- 172.21.15.28:0/2424547308 wait complete. 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 -- start start 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 Processor -- start 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 0x562b2a59de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a59e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44a400fca0 con 0x562b2a1c64d0 2024-09-21T05:51:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44a400f5a0 con 0x562b2a1be2f0 2024-09-21T05:51:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44a400c780 con 0x562b2a1e71b0 2024-09-21T05:51:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.214+0000 7f44d17dd740 10 client.0 _ll_register_callbacks cb 0x7f44b4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:38386/0 (socket says 172.21.15.28:38386) 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 -- 172.21.15.28:0/1752387840 learned_addr learned my addr 172.21.15.28:0/1752387840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c37fe640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a59e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 0x562b2a59de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 -- 172.21.15.28:0/1752387840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 msgr2=0x562b2a59de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 0x562b2a59de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 -- 172.21.15.28:0/1752387840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 msgr2=0x562b2a59e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a59e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c8d2d640 1 -- 172.21.15.28:0/1752387840 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2a60d420 con 0x562b2a1e71b0 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.215+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 0x562b2a59de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.216+0000 7f44c8d2d640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009a00 tx=0x7f44bc0b6040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934830e583e5f575 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.216+0000 7f44a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.216+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc00cd40 con 0x562b2a1e71b0 2024-09-21T05:51:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.216+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44ac031c90 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.216+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44bc0abb90 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.217+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f44bc0b1360 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.217+0000 7f44d17dd740 10 client.26230 fetch_fsmap learned FSMap version 38 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.217+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562b2a614760 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.217+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44ac031c90 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f44bc0d3070 con 0x562b2a1e71b0 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 10 client.26230 fetch_fsmap finished waiting for FSMap version 38 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 10 client.26230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 20 client.26230 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a17b1c0 0x562b2a6169a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 4 client.26230 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:51:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.218+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b2a4ff0f0 con 0x562b2a17b1c0 2024-09-21T05:51:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.219+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a17b1c0 0x562b2a6169a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.220+0000 7f44c3fff640 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a17b1c0 0x562b2a6169a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7556c5d952553dfa server_cookie=bd1481cd265c7d0b in_seq=0 out_seq=0 2024-09-21T05:51:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.220+0000 7f44a37fe640 10 client.26230 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.247+0000 7f44a37fe640 1 -- 172.21.15.28:0/1752387840 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b2a4ff0f0 con 0x562b2a17b1c0 2024-09-21T05:51:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.247+0000 7f44a37fe640 10 client.26230 handle_command_reply: tid=0 2024-09-21T05:51:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.247+0000 7f44d17dd740 1 client.26230 shutdown 2024-09-21T05:51:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.249+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 msgr2=0x562b2a60cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.249+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009a00 tx=0x7f44bc0b6040 comp rx=0 tx=0).stop 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 shutdown_connections 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562b2a17b1c0 0x562b2a6169a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b2a1e71b0 0x562b2a60cd80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b2a1c64d0 0x562b2a59e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 --2- 172.21.15.28:0/1752387840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b2a1be2f0 0x562b2a59de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 >> 172.21.15.28:0/1752387840 conn(0x562b2a1db300 msgr2=0x562b2a588ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.250+0000 7f44a37fe640 0 client.26230 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:13.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.252+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 shutdown_connections 2024-09-21T05:51:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.252+0000 7f44d17dd740 1 -- 172.21.15.28:0/1752387840 wait complete. 2024-09-21T05:51:13.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.253+0000 7f44d17dd740 20 client.26230 trim_cache size 0 max 16384 2024-09-21T05:51:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:51:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:51:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:51:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.255+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 msgr2=0x7f44b00b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.255+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 0x7f44b00b8cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009730 tx=0x7f44b40096c0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.255+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 msgr2=0x7f44c406f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.255+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b970 tx=0x7f44bc00be30 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 shutdown_connections 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44c4114510 0x7f44c406f900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b00b67b0 0x7f44b00b8cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44c4102140 0x7f44c406f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 --2- 172.21.15.28:0/2183107383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44c41017f0 0x7f44c4074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.257+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 >> 172.21.15.28:0/2183107383 conn(0x7f44c4077770 msgr2=0x7f44c41032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.259+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 shutdown_connections 2024-09-21T05:51:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:13.259+0000 7f44ca790640 1 -- 172.21.15.28:0/2183107383 wait complete. 2024-09-21T05:51:13.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:13 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:13 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:13.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:13 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:14 smithi181 ceph-mon[34819]: pgmap v451: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 46 MiB/s wr, 9.96k op/s 2024-09-21T05:51:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:14 smithi028 ceph-mon[31227]: pgmap v451: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 46 MiB/s wr, 9.96k op/s 2024-09-21T05:51:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:14 smithi129 ceph-mon[34885]: pgmap v451: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 46 MiB/s wr, 9.96k op/s 2024-09-21T05:51:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:16 smithi181 ceph-mon[34819]: pgmap v452: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-09-21T05:51:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:16 smithi028 ceph-mon[31227]: pgmap v452: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-09-21T05:51:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:16 smithi129 ceph-mon[34885]: pgmap v452: 97 pgs: 97 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-09-21T05:51:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:18 smithi028 ceph-mon[31227]: pgmap v453: 97 pgs: 97 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 31 MiB/s wr, 6.74k op/s 2024-09-21T05:51:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:18 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:18.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:18 smithi129 ceph-mon[34885]: pgmap v453: 97 pgs: 97 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 31 MiB/s wr, 6.74k op/s 2024-09-21T05:51:18.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:18 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:18.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:18 smithi181 ceph-mon[34819]: pgmap v453: 97 pgs: 97 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 31 MiB/s wr, 6.74k op/s 2024-09-21T05:51:18.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:18 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:20.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:20 smithi129 ceph-mon[34885]: pgmap v454: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.93k op/s 2024-09-21T05:51:20.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:20 smithi181 ceph-mon[34819]: pgmap v454: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.93k op/s 2024-09-21T05:51:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:20 smithi028 ceph-mon[31227]: pgmap v454: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.93k op/s 2024-09-21T05:51:22.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:22 smithi129 ceph-mon[34885]: pgmap v455: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 6.58k op/s 2024-09-21T05:51:22.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:22 smithi028 ceph-mon[31227]: pgmap v455: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 6.58k op/s 2024-09-21T05:51:22.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:22 smithi181 ceph-mon[34819]: pgmap v455: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 6.58k op/s 2024-09-21T05:51:24.345 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:24 smithi181 ceph-mon[34819]: pgmap v456: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 7.84k op/s 2024-09-21T05:51:24.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:24 smithi129 ceph-mon[34885]: pgmap v456: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 7.84k op/s 2024-09-21T05:51:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:24 smithi028 ceph-mon[31227]: pgmap v456: 97 pgs: 97 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 7.84k op/s 2024-09-21T05:51:26.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:26 smithi129 ceph-mon[34885]: pgmap v457: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 51 MiB/s wr, 11.51k op/s 2024-09-21T05:51:26.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.438+0000 7f3d5affd640 1 -- 172.21.15.28:0/1745171427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500318a0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.439+0000 7f3d616b0640 1 -- 172.21.15.28:0/1745171427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c111ca0 msgr2=0x7f3d5c10a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.439+0000 7f3d616b0640 1 --2- 172.21.15.28:0/1745171427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c10a3c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009900 tx=0x7f3d5002a440 comp rx=0 tx=0).stop 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.444+0000 7f3d616b0640 1 -- 172.21.15.28:0/1745171427 shutdown_connections 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.444+0000 7f3d616b0640 1 --2- 172.21.15.28:0/1745171427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c11cba0 0x7f3d5c11efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.444+0000 7f3d616b0640 1 --2- 172.21.15.28:0/1745171427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d5c10a900 0x7f3d5c10ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.444+0000 7f3d616b0640 1 --2- 172.21.15.28:0/1745171427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c10a3c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.444+0000 7f3d616b0640 1 -- 172.21.15.28:0/1745171427 >> 172.21.15.28:0/1745171427 conn(0x7f3d5c070c00 msgr2=0x7f3d5c071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.447+0000 7f3d616b0640 1 -- 172.21.15.28:0/1745171427 shutdown_connections 2024-09-21T05:51:26.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.449+0000 7f3d616b0640 1 -- 172.21.15.28:0/1745171427 wait complete. 2024-09-21T05:51:26.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.450+0000 7f3d616b0640 1 Processor -- start 2024-09-21T05:51:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.451+0000 7f3d616b0640 1 -- start start 2024-09-21T05:51:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.452+0000 7f3d616b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 0x7f3d5c11c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.453+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 0x7f3d5c11c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.453+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 0x7f3d5c11c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36404/0 (socket says 172.21.15.28:36404) 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.454+0000 7f3d616b0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c116410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d616b0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d5c11cba0 0x7f3d5c116950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d616b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c112800 con 0x7f3d5c10a900 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d616b0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c112680 con 0x7f3d5c11cba0 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d616b0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c112980 con 0x7f3d5c111ca0 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d5bfff640 1 -- 172.21.15.28:0/106989957 learned_addr learned my addr 172.21.15.28:0/106989957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:26.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.455+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c116410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.457+0000 7f3d5b7fe640 1 -- 172.21.15.28:0/106989957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d5c11cba0 msgr2=0x7f3d5c116950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:51:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.457+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d5c11cba0 0x7f3d5c116950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.457+0000 7f3d5b7fe640 1 -- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 msgr2=0x7f3d5c11c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.457+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 0x7f3d5c11c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.457+0000 7f3d5b7fe640 1 -- 172.21.15.28:0/106989957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c116ee0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.459+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c116410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009fd0 tx=0x7f3d4c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.460+0000 7f3d597fa640 1 -- 172.21.15.28:0/106989957 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c015020 con 0x7f3d5c111ca0 2024-09-21T05:51:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.460+0000 7f3d597fa640 1 -- 172.21.15.28:0/106989957 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4c011ce0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.460+0000 7f3d597fa640 1 -- 172.21.15.28:0/106989957 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c010de0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.462+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c1171d0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.462+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c1ad320 con 0x7f3d5c111ca0 2024-09-21T05:51:26.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.469+0000 7f3d69878740 1 Processor -- start 2024-09-21T05:51:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 -- start start 2024-09-21T05:51:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021fe4d0 0x5638025c5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638025396f0 con 0x5638021f62f0 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563802528b20 con 0x5638021fe4d0 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d69878740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56380252c930 con 0x56380221f1b0 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d60eaf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d60eaf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:60296/0 (socket says 172.21.15.28:60296) 2024-09-21T05:51:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.470+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2124403994 learned_addr learned my addr 172.21.15.28:0/2124403994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2124403994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021fe4d0 msgr2=0x5638025c5620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:51:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021fe4d0 0x5638025c5620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2124403994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 msgr2=0x5638025c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.472+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2124403994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638024ee530 con 0x56380221f1b0 2024-09-21T05:51:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.474+0000 7f3d597fa640 1 -- 172.21.15.28:0/106989957 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3d4c018020 con 0x7f3d5c111ca0 2024-09-21T05:51:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.475+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009d50 tx=0x7f3d5400b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cfcc9f69c6de0ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.476+0000 7f3d597fa640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 0x7f3d440c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.476+0000 7f3d597fa640 1 -- 172.21.15.28:0/106989957 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3d4c0e6ce0 con 0x7f3d5c111ca0 2024-09-21T05:51:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.478+0000 7f3d427fc640 1 -- 172.21.15.28:0/2124403994 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5400bd40 con 0x56380221f1b0 2024-09-21T05:51:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.478+0000 7f3d427fc640 1 -- 172.21.15.28:0/2124403994 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x56380221f1b0 2024-09-21T05:51:26.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d427fc640 1 -- 172.21.15.28:0/2124403994 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540115f0 con 0x56380221f1b0 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 -- 172.21.15.28:0/2124403994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 msgr2=0x5638025c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009d50 tx=0x7f3d5400b420 comp rx=0 tx=0).stop 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 -- 172.21.15.28:0/2124403994 shutdown_connections 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021fe4d0 0x5638025c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 --2- 172.21.15.28:0/2124403994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56380221f1b0 0x5638025c4ba0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d69878740 1 -- 172.21.15.28:0/2124403994 >> 172.21.15.28:0/2124403994 conn(0x563802213300 msgr2=0x5638025c2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.479+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 0x7f3d440c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.480+0000 7f3d69878740 1 -- 172.21.15.28:0/2124403994 shutdown_connections 2024-09-21T05:51:26.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.481+0000 7f3d69878740 1 -- 172.21.15.28:0/2124403994 wait complete. 2024-09-21T05:51:26.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.481+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 0x7f3d440c9d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5638025c3820 tx=0x7f3d50005890 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.481+0000 7f3d69878740 1 -- start start 2024-09-21T05:51:26.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.481+0000 7f3d69878740 1 Processor -- start 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563802460ab0 con 0x5638021f62f0 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5638024ce730 con 0x56380221f1b0 2024-09-21T05:51:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5638025d3370 con 0x5638021fe4d0 2024-09-21T05:51:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d69878740 10 client.0 _ll_register_callbacks cb 0x5638025226b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36416/0 (socket says 172.21.15.28:36416) 2024-09-21T05:51:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.482+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1772003654 learned_addr learned my addr 172.21.15.28:0/1772003654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 msgr2=0x5638025d0800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d0800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1772003654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 msgr2=0x5638025dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1772003654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380228a790 con 0x5638021f62f0 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.483+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3d50032d50 tx=0x7f3d50009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76923ec99c69049d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5004c070 con 0x5638021f62f0 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d54017070 con 0x5638021f62f0 2024-09-21T05:51:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3d50003b80 con 0x5638021f62f0 2024-09-21T05:51:26.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3d50042070 con 0x5638021f62f0 2024-09-21T05:51:26.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d69878740 10 client.16644 fetch_fsmap learned FSMap version 39 2024-09-21T05:51:26.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.484+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563802303620 con 0x5638021f62f0 2024-09-21T05:51:26.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.485+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x5638021f62f0 2024-09-21T05:51:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.486+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f3d50058020 con 0x5638021f62f0 2024-09-21T05:51:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.486+0000 7f3d69878740 10 client.16644 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T05:51:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d69878740 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:26.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d69878740 20 client.16644 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:26.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021ddd80 0x5638025e2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d69878740 4 client.16644 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:26.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380253b640 con 0x5638021ddd80 2024-09-21T05:51:26.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.487+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021ddd80 0x5638025e2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.489+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021ddd80 0x5638025e2910 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0259ff4cbf3485 server_cookie=b93c0093bfa6b1fa in_seq=0 out_seq=0 2024-09-21T05:51:26.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.489+0000 7f3d3b7fe640 10 client.16644 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:26.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.490+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1772003654 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56380253b640 con 0x5638021ddd80 2024-09-21T05:51:26.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.490+0000 7f3d3b7fe640 10 client.16644 handle_command_reply: tid=0 2024-09-21T05:51:26.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.490+0000 7f3d69878740 1 client.16644 shutdown 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 msgr2=0x5638025d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3d50032d50 tx=0x7f3d50009260 comp rx=0 tx=0).stop 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 shutdown_connections 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021ddd80 0x5638025e2910 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 --2- 172.21.15.28:0/1772003654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021f62f0 0x5638025d02c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 >> 172.21.15.28:0/1772003654 conn(0x563802213300 msgr2=0x5638025c5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d3b7fe640 0 client.16644 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:26.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 shutdown_connections 2024-09-21T05:51:26.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.491+0000 7f3d69878740 1 -- 172.21.15.28:0/1772003654 wait complete. 2024-09-21T05:51:26.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.492+0000 7f3d69878740 20 client.16644 trim_cache size 0 max 16384 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 Processor -- start 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 -- start start 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 0x5638025c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b5b20 con 0x5638021fe4d0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b76f0 con 0x56380221f1b0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d69878740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b61f0 con 0x5638021f62f0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:60306/0 (socket says 172.21.15.28:60306) 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d5bfff640 1 -- 172.21.15.28:0/2692202514 learned_addr learned my addr 172.21.15.28:0/2692202514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 0x5638025c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.518+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2692202514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 msgr2=0x5638025d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2692202514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 msgr2=0x5638025c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 0x5638025c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 -- 172.21.15.28:0/2692202514 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638021ddfb0 con 0x56380221f1b0 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 0x5638025c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:26.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc5b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5638025d2ef0 tx=0x7f3d54009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc37884c12cffdfe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d41ffb640 1 -- 172.21.15.28:0/2692202514 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54021020 con 0x56380221f1b0 2024-09-21T05:51:26.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.519+0000 7f3d41ffb640 1 -- 172.21.15.28:0/2692202514 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x56380221f1b0 2024-09-21T05:51:26.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d41ffb640 1 -- 172.21.15.28:0/2692202514 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540127a0 con 0x56380221f1b0 2024-09-21T05:51:26.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 -- 172.21.15.28:0/2692202514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 msgr2=0x5638025dc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc5b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5638025d2ef0 tx=0x7f3d54009ad0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 -- 172.21.15.28:0/2692202514 shutdown_connections 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56380221f1b0 0x5638025dc5b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638021fe4d0 0x5638025c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 --2- 172.21.15.28:0/2692202514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021f62f0 0x5638025d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 -- 172.21.15.28:0/2692202514 >> 172.21.15.28:0/2692202514 conn(0x563802213300 msgr2=0x5638025c23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 -- 172.21.15.28:0/2692202514 shutdown_connections 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.520+0000 7f3d69878740 1 -- 172.21.15.28:0/2692202514 wait complete. 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.521+0000 7f3d69878740 1 -- start start 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.521+0000 7f3d69878740 1 Processor -- start 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.521+0000 7f3d69878740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56380221f1b0 0x5638025d3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:32808/0 (socket says 172.21.15.28:32808) 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b7020 con 0x56380221f1b0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b6920 con 0x5638021f62f0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1b3bb0 con 0x5638021fe4d0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1489472996 learned_addr learned my addr 172.21.15.28:0/1489472996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d69878740 10 client.0 _ll_register_callbacks cb 0x7f3d50032ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d60eaf640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56380221f1b0 0x5638025d3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 msgr2=0x5638025d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1489472996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56380221f1b0 msgr2=0x5638025d3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56380221f1b0 0x5638025d3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.522+0000 7f3d5bfff640 1 -- 172.21.15.28:0/1489472996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638025d42e0 con 0x5638021f62f0 2024-09-21T05:51:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d5bfff640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d5003c530 tx=0x7f3d5002adf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1f53a0dedd00cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50060070 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3d5003cd90 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d54017070 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3d5004c040 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d69878740 10 client.26225 fetch_fsmap learned FSMap version 39 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56380264c270 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.523+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f3d5006d030 con 0x5638021f62f0 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 10 client.26225 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 10 client.26225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 20 client.26225 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021b31c0 0x56380264ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 4 client.26225 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:51:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638025e2f00 con 0x5638021b31c0 2024-09-21T05:51:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.524+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021b31c0 0x56380264ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.525+0000 7f3d5b7fe640 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021b31c0 0x56380264ea20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cc5745bfb9f67d1 server_cookie=4ce16f341513167d in_seq=0 out_seq=0 2024-09-21T05:51:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.525+0000 7f3d3b7fe640 10 client.26225 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:26.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:26 smithi181 ceph-mon[34819]: pgmap v457: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 51 MiB/s wr, 11.51k op/s 2024-09-21T05:51:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.550+0000 7f3d3b7fe640 1 -- 172.21.15.28:0/1489472996 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638025e2f00 con 0x5638021b31c0 2024-09-21T05:51:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.550+0000 7f3d3b7fe640 10 client.26225 handle_command_reply: tid=0 2024-09-21T05:51:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.550+0000 7f3d69878740 1 client.26225 shutdown 2024-09-21T05:51:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 msgr2=0x5638025d8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d5003c530 tx=0x7f3d5002adf0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 shutdown_connections 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5638021b31c0 0x56380264ea20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56380221f1b0 0x5638025d3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d3b7fe640 0 client.26225 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5638021fe4d0 0x5638025d35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 --2- 172.21.15.28:0/1489472996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5638021f62f0 0x5638025d8e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 >> 172.21.15.28:0/1489472996 conn(0x563802213300 msgr2=0x5638025c1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 shutdown_connections 2024-09-21T05:51:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.551+0000 7f3d69878740 1 -- 172.21.15.28:0/1489472996 wait complete. 2024-09-21T05:51:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.552+0000 7f3d69878740 20 client.26225 trim_cache size 0 max 16384 2024-09-21T05:51:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:51:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:51:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:51:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:51:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.554+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 msgr2=0x7f3d440c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.554+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 0x7f3d440c9d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5638025c3820 tx=0x7f3d50005890 comp rx=0 tx=0).stop 2024-09-21T05:51:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 msgr2=0x7f3d5c116410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c116410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009fd0 tx=0x7f3d4c00b870 comp rx=0 tx=0).stop 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 shutdown_connections 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d5c11cba0 0x7f3d5c116950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d5c111ca0 0x7f3d5c116410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3d440c7860 0x7f3d440c9d70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 --2- 172.21.15.28:0/106989957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d5c10a900 0x7f3d5c11c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 >> 172.21.15.28:0/106989957 conn(0x7f3d5c070c00 msgr2=0x7f3d5c110920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 shutdown_connections 2024-09-21T05:51:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:26.555+0000 7f3d616b0640 1 -- 172.21.15.28:0/106989957 wait complete. 2024-09-21T05:51:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:26 smithi028 ceph-mon[31227]: pgmap v457: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 51 MiB/s wr, 11.51k op/s 2024-09-21T05:51:28.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:28 smithi129 ceph-mon[34885]: pgmap v458: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 50 MiB/s wr, 11.21k op/s 2024-09-21T05:51:28.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:28 smithi181 ceph-mon[34819]: pgmap v458: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 50 MiB/s wr, 11.21k op/s 2024-09-21T05:51:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:28 smithi028 ceph-mon[31227]: pgmap v458: 97 pgs: 97 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 50 MiB/s wr, 11.21k op/s 2024-09-21T05:51:30.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:30 smithi129 ceph-mon[34885]: pgmap v459: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 56 MiB/s wr, 12.63k op/s 2024-09-21T05:51:30.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:30 smithi181 ceph-mon[34819]: pgmap v459: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 56 MiB/s wr, 12.63k op/s 2024-09-21T05:51:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:30 smithi028 ceph-mon[31227]: pgmap v459: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 56 MiB/s wr, 12.63k op/s 2024-09-21T05:51:31.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:51:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.128+0000 7f8ab959a640 1 -- 172.21.15.28:0/1916463467 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4031e60 con 0x7f8abc11cc30 2024-09-21T05:51:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.128+0000 7f8ac0929640 1 -- 172.21.15.28:0/1916463467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc11cc30 msgr2=0x7f8abc11f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.128+0000 7f8ac0929640 1 --2- 172.21.15.28:0/1916463467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc11cc30 0x7f8abc11f140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009ca0 tx=0x7f8ab402a8f0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 -- 172.21.15.28:0/1916463467 shutdown_connections 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 --2- 172.21.15.28:0/1916463467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc11cc30 0x7f8abc11f140 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 --2- 172.21.15.28:0/1916463467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc10a9e0 0x7f8abc10adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 --2- 172.21.15.28:0/1916463467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc111f40 0x7f8abc10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 -- 172.21.15.28:0/1916463467 >> 172.21.15.28:0/1916463467 conn(0x7f8abc070ec0 msgr2=0x7f8abc0712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 -- 172.21.15.28:0/1916463467 shutdown_connections 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.129+0000 7f8ac0929640 1 -- 172.21.15.28:0/1916463467 wait complete. 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 Processor -- start 2024-09-21T05:51:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 -- start start 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 0x7f8abc1a9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 0x7f8abc1a97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc122900 con 0x7f8abc11cc30 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc122780 con 0x7f8abc111f40 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8ac0929640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc122a80 con 0x7f8abc10a9e0 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8abad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8abad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36458/0 (socket says 172.21.15.28:36458) 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.131+0000 7f8abad9d640 1 -- 172.21.15.28:0/2460773348 learned_addr learned my addr 172.21.15.28:0/2460773348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 0x7f8abc1a97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 0x7f8abc1a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 -- 172.21.15.28:0/2460773348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 msgr2=0x7f8abc1a9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 0x7f8abc1a9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 -- 172.21.15.28:0/2460773348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 msgr2=0x7f8abc1a97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 0x7f8abc1a97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 -- 172.21.15.28:0/2460773348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc1af0c0 con 0x7f8abc11cc30 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 0x7f8abc1a97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 0x7f8abc1a9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.132+0000 7f8abad9d640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8ab402ae60 tx=0x7f8ab40316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.133+0000 7f8aab7fe640 1 -- 172.21.15.28:0/2460773348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4003bf0 con 0x7f8abc11cc30 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.133+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc1af350 con 0x7f8abc11cc30 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.133+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc1b5bd0 con 0x7f8abc11cc30 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.133+0000 7f8aab7fe640 1 -- 172.21.15.28:0/2460773348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ab402d070 con 0x7f8abc11cc30 2024-09-21T05:51:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.135+0000 7f8aab7fe640 1 -- 172.21.15.28:0/2460773348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4030850 con 0x7f8abc11cc30 2024-09-21T05:51:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.137+0000 7f8ac6e9d740 1 Processor -- start 2024-09-21T05:51:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.138+0000 7f8aab7fe640 1 -- 172.21.15.28:0/2460773348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8ab40309f0 con 0x7f8abc11cc30 2024-09-21T05:51:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 -- start start 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 0x564415f55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564415ecbe50 con 0x564415b8d390 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564415ebb280 con 0x564415b851b0 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564415ebf090 con 0x564415bae070 2024-09-21T05:51:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8aab7fe640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 0x7f8aa40b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:60352/0 (socket says 172.21.15.28:60352) 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.140+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2784419241 learned_addr learned my addr 172.21.15.28:0/2784419241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 0x564415f55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8aab7fe640 1 -- 172.21.15.28:0/2460773348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8ab4108a20 con 0x7f8abc11cc30 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8abad9d640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 0x7f8aa40b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2784419241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 msgr2=0x564415f55b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 0x564415f55b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2784419241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 msgr2=0x564415f560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2784419241 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564415c19650 con 0x564415bae070 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 0x564415f55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.141+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00058c0 tx=0x7f8ab000b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291ae23523872fc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.142+0000 7f8aa8ff9640 1 -- 172.21.15.28:0/2784419241 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0004250 con 0x564415bae070 2024-09-21T05:51:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.142+0000 7f8aa8ff9640 1 -- 172.21.15.28:0/2784419241 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ab00043f0 con 0x564415bae070 2024-09-21T05:51:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.142+0000 7f8aa8ff9640 1 -- 172.21.15.28:0/2784419241 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0003830 con 0x564415bae070 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2784419241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 msgr2=0x564415f55650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00058c0 tx=0x7f8ab000b4e0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8abad9d640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 0x7f8aa40b8c70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006010 tx=0x7f8ab40095c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2784419241 shutdown_connections 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b851b0 0x564415f55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2784419241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f55650 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.143+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2784419241 >> 172.21.15.28:0/2784419241 conn(0x564415ba2460 msgr2=0x564415f53540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.144+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2784419241 shutdown_connections 2024-09-21T05:51:32.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.144+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2784419241 wait complete. 2024-09-21T05:51:32.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.145+0000 7f8ac6e9d740 1 -- start start 2024-09-21T05:51:32.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.145+0000 7f8ac6e9d740 1 Processor -- start 2024-09-21T05:51:32.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b8d390 0x564415f611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36472/0 (socket says 172.21.15.28:36472) 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.146+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2751672123 learned_addr learned my addr 172.21.15.28:0/2751672123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564415d1c700 con 0x564415b851b0 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564415e56110 con 0x564415b8d390 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564415f63ed0 con 0x564415bae070 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8ac6e9d740 10 client.0 _ll_register_callbacks cb 0x564415e6f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b8d390 0x564415f611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.147+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 msgr2=0x564415f616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2751672123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b8d390 msgr2=0x564415f611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b8d390 0x564415f611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564415e83c40 con 0x564415b851b0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00043a0 tx=0x7f8ab0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe6cb2c92b08d73 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.148+0000 7f8aa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0028020 con 0x564415b851b0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8ab000eb60 con 0x564415b851b0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ab00043f0 con 0x564415b851b0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8ab00035f0 con 0x564415b851b0 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8ac6e9d740 10 client.16659 fetch_fsmap learned FSMap version 39 2024-09-21T05:51:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564415f6fb30 con 0x564415b851b0 2024-09-21T05:51:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.149+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ab00043f0 con 0x564415b851b0 2024-09-21T05:51:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f8ab0011780 con 0x564415b851b0 2024-09-21T05:51:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 10 client.16659 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T05:51:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 20 client.16659 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415b6c9f0 0x564415f72fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 4 client.16659 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.150+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564415ecdda0 con 0x564415b6c9f0 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.151+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415b6c9f0 0x564415f72fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.151+0000 7f8aa1ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.151+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415b6c9f0 0x564415f72fa0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f964635875f5642 server_cookie=5b80f24a6f438d7e in_seq=0 out_seq=0 2024-09-21T05:51:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.152+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2751672123 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564415ecdda0 con 0x564415b6c9f0 2024-09-21T05:51:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.152+0000 7f8aa1ffb640 10 client.16659 handle_command_reply: tid=0 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.152+0000 7f8ac6e9d740 1 client.16659 shutdown 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 msgr2=0x564415f60c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00043a0 tx=0x7f8ab0005dc0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 shutdown_connections 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415b6c9f0 0x564415f72fa0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415bae070 0x564415f616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b8d390 0x564415f611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2751672123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b851b0 0x564415f60c70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 >> 172.21.15.28:0/2751672123 conn(0x564415ba2460 msgr2=0x564415f587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.153+0000 7f8aa1ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.154+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 shutdown_connections 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.154+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2751672123 wait complete. 2024-09-21T05:51:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.154+0000 7f8ac6e9d740 20 client.16659 trim_cache size 0 max 16384 2024-09-21T05:51:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 Processor -- start 2024-09-21T05:51:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 -- start start 2024-09-21T05:51:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f628a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415bae070 0x564415f5de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c00e830 con 0x564415b8d390 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c010400 con 0x564415bae070 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.186+0000 7f8ac6e9d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c00ef00 con 0x564415b851b0 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415bae070 0x564415f5de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:60362/0 (socket says 172.21.15.28:60362) 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/1638202666 learned_addr learned my addr 172.21.15.28:0/1638202666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/1638202666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415bae070 msgr2=0x564415f5de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415bae070 0x564415f5de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/1638202666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 msgr2=0x564415f628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/1638202666 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564415f61d10 con 0x564415b851b0 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.187+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000bc40 tx=0x7f8ab0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=384ec6201628624e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.188+0000 7f8aa3fff640 1 -- 172.21.15.28:0/1638202666 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0028070 con 0x564415b851b0 2024-09-21T05:51:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.188+0000 7f8aa3fff640 1 -- 172.21.15.28:0/1638202666 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ab000b980 con 0x564415b851b0 2024-09-21T05:51:32.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.188+0000 7f8aa3fff640 1 -- 172.21.15.28:0/1638202666 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00039f0 con 0x564415b851b0 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.189+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/1638202666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 msgr2=0x564415f57160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.189+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000bc40 tx=0x7f8ab0000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/1638202666 shutdown_connections 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415bae070 0x564415f5de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415b8d390 0x564415f628a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/1638202666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b851b0 0x564415f57160 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000bc40 tx=0x7f8ab0000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/1638202666 >> 172.21.15.28:0/1638202666 conn(0x564415ba2460 msgr2=0x564415f62190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/1638202666 shutdown_connections 2024-09-21T05:51:32.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.190+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/1638202666 wait complete. 2024-09-21T05:51:32.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.191+0000 7f8ac6e9d740 1 -- start start 2024-09-21T05:51:32.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.191+0000 7f8ac6e9d740 1 Processor -- start 2024-09-21T05:51:32.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.192+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 0x564415f69ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.192+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 0x564415f64fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.192+0000 7f8ac6e9d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b41da0 0x564415f655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.192+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 0x564415f69ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.192+0000 7f8ab9d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 0x564415f69ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:60364/0 (socket says 172.21.15.28:60364) 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8ab9d9b640 1 -- 172.21.15.28:0/2635669475 learned_addr learned my addr 172.21.15.28:0/2635669475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 0x564415f64fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c00fd30 con 0x564415bae070 2024-09-21T05:51:32.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c00f630 con 0x564415b41da0 2024-09-21T05:51:32.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c00c810 con 0x564415b8d390 2024-09-21T05:51:32.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b41da0 0x564415f655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8aba59c640 1 -- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 msgr2=0x564415f69ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 0x564415f69ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8aba59c640 1 -- 172.21.15.28:0/2635669475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b41da0 msgr2=0x564415f655b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.193+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b41da0 0x564415f655b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8aba59c640 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564415f65d40 con 0x564415bae070 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8ac6e9d740 10 client.0 _ll_register_callbacks cb 0x7f8ab000bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8aba59c640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 0x564415f64fb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x564415f64100 tx=0x7f8aac014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76842fdd66b5f057 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ab000b980 con 0x564415bae070 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8aa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.194+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2635669475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac014e20 con 0x564415bae070 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.195+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2635669475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8aac012450 con 0x564415bae070 2024-09-21T05:51:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.195+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2635669475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8aac0126d0 con 0x564415bae070 2024-09-21T05:51:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.195+0000 7f8ac6e9d740 10 client.16662 fetch_fsmap learned FSMap version 39 2024-09-21T05:51:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.195+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=39}) -- 0x564415fdd6d0 con 0x564415bae070 2024-09-21T05:51:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.195+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2635669475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f8aac012870 con 0x564415bae070 2024-09-21T05:51:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 10 client.16662 fetch_fsmap finished waiting for FSMap version 39 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 20 client.16662 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415fddc90 0x564415fe0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 4 client.16662 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564415f53630 con 0x564415fddc90 2024-09-21T05:51:32.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.196+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415fddc90 0x564415fe0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.197+0000 7f8ab9d9b640 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415fddc90 0x564415fe0170 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ade6c2b15a7f17a server_cookie=88144599b57c01ff in_seq=0 out_seq=0 2024-09-21T05:51:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.197+0000 7f8aa1ffb640 10 client.16662 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8aa1ffb640 1 -- 172.21.15.28:0/2635669475 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2078 (crc 0 0 0) 0x564415f53630 con 0x564415fddc90 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8aa1ffb640 10 client.16662 handle_command_reply: tid=0 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 client.16662 shutdown 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 msgr2=0x564415f64fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 0x564415f64fb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x564415f64100 tx=0x7f8aac014840 comp rx=0 tx=0).stop 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 shutdown_connections 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564415fddc90 0x564415fe0170 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564415b41da0 0x564415f655b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564415bae070 0x564415f64fb0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 --2- 172.21.15.28:0/2635669475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564415b8d390 0x564415f69ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 >> 172.21.15.28:0/2635669475 conn(0x564415ba2460 msgr2=0x564415f62a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.198+0000 7f8aa1ffb640 0 client.16662 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.199+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 shutdown_connections 2024-09-21T05:51:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.199+0000 7f8ac6e9d740 1 -- 172.21.15.28:0/2635669475 wait complete. 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.199+0000 7f8ac6e9d740 20 client.16662 trim_cache size 0 max 16384 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:51:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:51:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:51:32.206 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 msgr2=0x7f8aa40b8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 0x7f8aa40b8c70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006010 tx=0x7f8ab40095c0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 msgr2=0x7f8abc1aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8ab402ae60 tx=0x7f8ab40316b0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 shutdown_connections 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8aa40b6760 0x7f8aa40b8c70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8abc11cc30 0x7f8abc1aea50 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8abc111f40 0x7f8abc1a97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 --2- 172.21.15.28:0/2460773348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8abc10a9e0 0x7f8abc1a9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.200+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 >> 172.21.15.28:0/2460773348 conn(0x7f8abc070ec0 msgr2=0x7f8abc11e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.201+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 shutdown_connections 2024-09-21T05:51:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:32.202+0000 7f8ac0929640 1 -- 172.21.15.28:0/2460773348 wait complete. 2024-09-21T05:51:32.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:32 smithi028 ceph-mon[31227]: pgmap v460: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 MiB/s wr, 12.07k op/s 2024-09-21T05:51:32.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:32 smithi129 ceph-mon[34885]: pgmap v460: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 MiB/s wr, 12.07k op/s 2024-09-21T05:51:32.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:32 smithi181 ceph-mon[34819]: pgmap v460: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 MiB/s wr, 12.07k op/s 2024-09-21T05:51:33.752 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:51:33.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:51:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:34 smithi028 ceph-mon[31227]: pgmap v461: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 53 MiB/s wr, 12.31k op/s 2024-09-21T05:51:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:34 smithi129 ceph-mon[34885]: pgmap v461: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 53 MiB/s wr, 12.31k op/s 2024-09-21T05:51:34.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:34 smithi181 ceph-mon[34819]: pgmap v461: 97 pgs: 97 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 53 MiB/s wr, 12.31k op/s 2024-09-21T05:51:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.989+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2180218218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 msgr2=0x7f98e407b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.989+0000 7f98ea9b7640 1 --2- 172.21.15.28:0/2180218218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e407b690 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00afd0 tx=0x7f98dc02e980 comp rx=0 tx=0).stop 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.991+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2180218218 shutdown_connections 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.991+0000 7f98ea9b7640 1 --2- 172.21.15.28:0/2180218218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e407b690 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.991+0000 7f98ea9b7640 1 --2- 172.21.15.28:0/2180218218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 0x7f98e410fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.991+0000 7f98ea9b7640 1 --2- 172.21.15.28:0/2180218218 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98e410b8e0 0x7f98e410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.991+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2180218218 >> 172.21.15.28:0/2180218218 conn(0x7f98e406e9f0 msgr2=0x7f98e406ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:34.998+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2180218218 shutdown_connections 2024-09-21T05:51:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.000+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2180218218 wait complete. 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 Processor -- start 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 -- start start 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98e410b8e0 0x7f98e4080560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 0x7f98e4080aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98e407ec60 con 0x7f98e410f8c0 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98e407eae0 con 0x7f98e4110210 2024-09-21T05:51:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.001+0000 7f98ea9b7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f98e407ede0 con 0x7f98e410b8e0 2024-09-21T05:51:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.002+0000 7f98ea1b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.002+0000 7f98ea1b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:32876/0 (socket says 172.21.15.28:32876) 2024-09-21T05:51:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.002+0000 7f98ea1b6640 1 -- 172.21.15.28:0/2360305515 learned_addr learned my addr 172.21.15.28:0/2360305515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.004+0000 7f98e91b4640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 0x7f98e4080aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.005+0000 7f98ea1b6640 1 -- 172.21.15.28:0/2360305515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98e410b8e0 msgr2=0x7f98e4080560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:51:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.005+0000 7f98ea1b6640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98e410b8e0 0x7f98e4080560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.005+0000 7f98ea1b6640 1 -- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 msgr2=0x7f98e4080aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.005+0000 7f98ea1b6640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 0x7f98e4080aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.005+0000 7f98ea1b6640 1 -- 172.21.15.28:0/2360305515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e41c60e0 con 0x7f98e4110210 2024-09-21T05:51:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.009+0000 7f98ea1b6640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98dc02eef0 tx=0x7f98dc034f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.010+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc012320 con 0x7f98e4110210 2024-09-21T05:51:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.010+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98dc0124c0 con 0x7f98e4110210 2024-09-21T05:51:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.010+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc033c40 con 0x7f98e4110210 2024-09-21T05:51:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.010+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2360305515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e41c6370 con 0x7f98e4110210 2024-09-21T05:51:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.010+0000 7f98ea9b7640 1 -- 172.21.15.28:0/2360305515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e41c6880 con 0x7f98e4110210 2024-09-21T05:51:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.020+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f98dc046020 con 0x7f98e4110210 2024-09-21T05:51:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.020+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98cc005290 con 0x7f98e4110210 2024-09-21T05:51:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.021+0000 7f98daffd640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 0x7f98d00b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.022+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f98dc033de0 con 0x7f98e4110210 2024-09-21T05:51:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.025+0000 7f98e99b5640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 0x7f98d00b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.032+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f98dc04d020 con 0x7f98e4110210 2024-09-21T05:51:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.035+0000 7f98e99b5640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 0x7f98d00b8ce0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f98e4085ff0 tx=0x7f98e0005b80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:35.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.447+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98cc005580 con 0x7f98e4110210 2024-09-21T05:51:35.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.448+0000 7f98daffd640 1 -- 172.21.15.28:0/2360305515 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11991 (secure 0 0 0) 0x7f98dc0f1020 con 0x7f98e4110210 2024-09-21T05:51:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:51:35.452 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"btime":"2024-09-21T05:51:17:039943+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:51:16.049543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:51:35.452 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 39 2024-09-21T05:51:35.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.453+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 msgr2=0x7f98d00b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:35.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 0x7f98d00b8ce0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f98e4085ff0 tx=0x7f98e0005b80 comp rx=0 tx=0).stop 2024-09-21T05:51:35.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 msgr2=0x7f98e4080fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:35.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98dc02eef0 tx=0x7f98dc034f80 comp rx=0 tx=0).stop 2024-09-21T05:51:35.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 shutdown_connections 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98e4110210 0x7f98e4080fe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.454+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f98d00b67d0 0x7f98d00b8ce0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.455+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e410f8c0 0x7f98e4080aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.455+0000 7f98d8ff9640 1 --2- 172.21.15.28:0/2360305515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98e410b8e0 0x7f98e4080560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.455+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 >> 172.21.15.28:0/2360305515 conn(0x7f98e406e9f0 msgr2=0x7f98e410d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.455+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 shutdown_connections 2024-09-21T05:51:35.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:35.455+0000 7f98d8ff9640 1 -- 172.21.15.28:0/2360305515 wait complete. 2024-09-21T05:51:36.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:36 smithi028 ceph-mon[31227]: pgmap v462: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 55 MiB/s wr, 12.72k op/s 2024-09-21T05:51:36.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:36 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2360305515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:51:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:36 smithi129 ceph-mon[34885]: pgmap v462: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 55 MiB/s wr, 12.72k op/s 2024-09-21T05:51:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:36 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2360305515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:51:36.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:51:36.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:36 smithi181 ceph-mon[34819]: pgmap v462: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 55 MiB/s wr, 12.72k op/s 2024-09-21T05:51:36.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:36 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2360305515' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:51:36.803 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:37.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2060278599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 msgr2=0x7f9c88104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2060278599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c88104cf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009900 tx=0x7f9c8402a3f0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2060278599 shutdown_connections 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2060278599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88105230 0x7f9c8810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2060278599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c88104cf0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2060278599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88102430 0x7f9c88102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2060278599 >> 172.21.15.28:0/2060278599 conn(0x7f9c88077800 msgr2=0x7f9c88077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2060278599 shutdown_connections 2024-09-21T05:51:37.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.531+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2060278599 wait complete. 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 Processor -- start 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 -- start start 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88102430 0x7f9c8819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c8819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88113aa0 con 0x7f9c881048e0 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88113920 con 0x7f9c88105230 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.532+0000 7f9c8fab9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88113c20 con 0x7f9c88102430 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39294/0 (socket says 172.21.15.28:39294) 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 -- 172.21.15.28:0/2309177848 learned_addr learned my addr 172.21.15.28:0/2309177848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8d855640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88102430 0x7f9c8819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8d054640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c8819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 -- 172.21.15.28:0/2309177848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88102430 msgr2=0x7f9c8819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88102430 0x7f9c8819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 -- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 msgr2=0x7f9c8819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c8819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.533+0000 7f9c8e056640 1 -- 172.21.15.28:0/2309177848 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c881a0440 con 0x7f9c88105230 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.534+0000 7f9c8e056640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000eb10 tx=0x7f9c8000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.534+0000 7f9c7effd640 1 -- 172.21.15.28:0/2309177848 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c80004b20 con 0x7f9c88105230 2024-09-21T05:51:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.534+0000 7f9c7effd640 1 -- 172.21.15.28:0/2309177848 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c80004cc0 con 0x7f9c88105230 2024-09-21T05:51:37.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.535+0000 7f9c7effd640 1 -- 172.21.15.28:0/2309177848 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c80010670 con 0x7f9c88105230 2024-09-21T05:51:37.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.535+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c881a0730 con 0x7f9c88105230 2024-09-21T05:51:37.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.535+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c881a6e90 con 0x7f9c88105230 2024-09-21T05:51:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.540+0000 7f9c7effd640 1 -- 172.21.15.28:0/2309177848 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9c80005020 con 0x7f9c88105230 2024-09-21T05:51:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.540+0000 7f9c9602d740 1 Processor -- start 2024-09-21T05:51:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.540+0000 7f9c9602d740 1 -- start start 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ecd1b0 0x55ff6329d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff6329dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff63213ea0 con 0x55ff62ed5390 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ff632032d0 con 0x55ff62ecd1b0 2024-09-21T05:51:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c9602d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ff632070e0 con 0x55ff62ef6070 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57260/0 (socket says 172.21.15.28:57260) 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.541+0000 7f9c8d054640 1 -- 172.21.15.28:0/110050525 learned_addr learned my addr 172.21.15.28:0/110050525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c7effd640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 0x7f9c740b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff6329dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8e056640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 0x7f9c740b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c7effd640 1 -- 172.21.15.28:0/2309177848 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9c800e7cb0 con 0x7f9c88105230 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 -- 172.21.15.28:0/110050525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ecd1b0 msgr2=0x55ff6329d800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ecd1b0 0x55ff6329d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 -- 172.21.15.28:0/110050525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 msgr2=0x55ff6329dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff6329dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.542+0000 7f9c8d054640 1 -- 172.21.15.28:0/110050525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff62f61650 con 0x55ff62ef6070 2024-09-21T05:51:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.543+0000 7f9c8d054640 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9c880756a0 tx=0x7f9c84000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=215954b95a2dcdfb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.543+0000 7f9c8e056640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 0x7f9c740b8c90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9c800e59c0 tx=0x7f9c800a2320 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.543+0000 7f9c73fff640 1 -- 172.21.15.28:0/110050525 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84044030 con 0x55ff62ef6070 2024-09-21T05:51:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.543+0000 7f9c73fff640 1 -- 172.21.15.28:0/110050525 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c840314a0 con 0x55ff62ef6070 2024-09-21T05:51:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.543+0000 7f9c73fff640 1 -- 172.21.15.28:0/110050525 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84035030 con 0x55ff62ef6070 2024-09-21T05:51:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.544+0000 7f9c9602d740 1 -- 172.21.15.28:0/110050525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 msgr2=0x55ff6329d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.544+0000 7f9c9602d740 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9c880756a0 tx=0x7f9c84000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 -- 172.21.15.28:0/110050525 shutdown_connections 2024-09-21T05:51:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff6329dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ecd1b0 0x55ff6329d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 --2- 172.21.15.28:0/110050525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ef6070 0x55ff6329d2c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 -- 172.21.15.28:0/110050525 >> 172.21.15.28:0/110050525 conn(0x55ff62eea460 msgr2=0x55ff6329b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.545+0000 7f9c9602d740 1 -- 172.21.15.28:0/110050525 shutdown_connections 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.546+0000 7f9c9602d740 1 -- 172.21.15.28:0/110050525 wait complete. 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.546+0000 7f9c9602d740 1 -- start start 2024-09-21T05:51:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.546+0000 7f9c9602d740 1 Processor -- start 2024-09-21T05:51:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ef6070 0x55ff632a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff63064700 con 0x55ff62ef6070 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ff6319e160 con 0x55ff62ed5390 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c9602d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ff632aba40 con 0x55ff62ecd1b0 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c8d855640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c8d855640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39320/0 (socket says 172.21.15.28:39320) 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.547+0000 7f9c8d855640 1 -- 172.21.15.28:0/1240146843 learned_addr learned my addr 172.21.15.28:0/1240146843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c9602d740 10 client.0 _ll_register_callbacks cb 0x55ff631b79e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ef6070 0x55ff632a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 -- 172.21.15.28:0/1240146843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 msgr2=0x55ff632a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632a88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 -- 172.21.15.28:0/1240146843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ef6070 msgr2=0x55ff632a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ef6070 0x55ff632a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 -- 172.21.15.28:0/1240146843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff631cbc90 con 0x55ff62ed5390 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800be80 tx=0x7f9c780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1d58fcb1059cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.548+0000 7f9c717fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c78011850 con 0x55ff62ed5390 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9c780119f0 con 0x55ff62ed5390 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c840314a0 con 0x55ff62ed5390 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f9c78011bf0 con 0x55ff62ed5390 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c9602d740 10 client.26255 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.549+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ff632b7710 con 0x55ff62ed5390 2024-09-21T05:51:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.550+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c840314a0 con 0x55ff62ed5390 2024-09-21T05:51:37.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f9c78024390 con 0x55ff62ed5390 2024-09-21T05:51:37.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 10 client.26255 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:37.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 10 client.26255 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:51:37.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 20 client.26255 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62eb49f0 0x55ff632bacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 4 client.26255 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff63215df0 con 0x55ff62eb49f0 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.551+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62eb49f0 0x55ff632bacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.552+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62eb49f0 0x55ff632bacb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c17a39b578f9492 server_cookie=4a02fb81eb918c18 in_seq=0 out_seq=0 2024-09-21T05:51:37.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.552+0000 7f9c717fa640 10 client.26255 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.553+0000 7f9c717fa640 1 -- 172.21.15.28:0/1240146843 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ff63215df0 con 0x55ff62eb49f0 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.553+0000 7f9c717fa640 10 client.26255 handle_command_reply: tid=0 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.553+0000 7f9c9602d740 1 client.26255 shutdown 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.553+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 msgr2=0x55ff632a8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.553+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800be80 tx=0x7f9c780107a0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 shutdown_connections 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62eb49f0 0x55ff632bacb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ef6070 0x55ff632a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8e20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c717fa640 0 client.26255 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1240146843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 >> 172.21.15.28:0/1240146843 conn(0x55ff62eea460 msgr2=0x55ff632a0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 shutdown_connections 2024-09-21T05:51:37.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 1 -- 172.21.15.28:0/1240146843 wait complete. 2024-09-21T05:51:37.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.554+0000 7f9c9602d740 20 client.26255 trim_cache size 0 max 16384 2024-09-21T05:51:37.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.578+0000 7f9c9602d740 1 Processor -- start 2024-09-21T05:51:37.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.578+0000 7f9c9602d740 1 -- start start 2024-09-21T05:51:37.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.578+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.578+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ecd1b0 0x55ff632a8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.578+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c9602d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881af7e0 con 0x55ff62ecd1b0 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c9602d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881b1380 con 0x55ff62ed5390 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c9602d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881afeb0 con 0x55ff62eb49f0 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57280/0 (socket says 172.21.15.28:57280) 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 -- 172.21.15.28:0/1634344161 learned_addr learned my addr 172.21.15.28:0/1634344161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ecd1b0 0x55ff632a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 -- 172.21.15.28:0/1634344161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 msgr2=0x55ff632a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 -- 172.21.15.28:0/1634344161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ecd1b0 msgr2=0x55ff632a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ecd1b0 0x55ff632a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.579+0000 7f9c8d054640 1 -- 172.21.15.28:0/1634344161 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff632a43e0 con 0x55ff62eb49f0 2024-09-21T05:51:37.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.580+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c84006010 tx=0x7f9c84032cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4da1130fdd5208da server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.580+0000 7f9c737fe640 1 -- 172.21.15.28:0/1634344161 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84035040 con 0x55ff62eb49f0 2024-09-21T05:51:37.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.580+0000 7f9c737fe640 1 -- 172.21.15.28:0/1634344161 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c840314a0 con 0x55ff62eb49f0 2024-09-21T05:51:37.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.580+0000 7f9c737fe640 1 -- 172.21.15.28:0/1634344161 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84004930 con 0x55ff62eb49f0 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 -- 172.21.15.28:0/1634344161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 msgr2=0x55ff632b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c84006010 tx=0x7f9c84032cf0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 -- 172.21.15.28:0/1634344161 shutdown_connections 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62ed5390 0x55ff632a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ecd1b0 0x55ff632a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1634344161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62eb49f0 0x55ff632b4c60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.581+0000 7f9c9602d740 1 -- 172.21.15.28:0/1634344161 >> 172.21.15.28:0/1634344161 conn(0x55ff62eea460 msgr2=0x55ff632a7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.582+0000 7f9c9602d740 1 -- 172.21.15.28:0/1634344161 shutdown_connections 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.582+0000 7f9c9602d740 1 -- 172.21.15.28:0/1634344161 wait complete. 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.582+0000 7f9c9602d740 1 -- start start 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.582+0000 7f9c9602d740 1 Processor -- start 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff632ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881b0cb0 con 0x55ff62ed5390 2024-09-21T05:51:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881b05b0 con 0x55ff62eb49f0 2024-09-21T05:51:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.583+0000 7f9c9602d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9c881ad7c0 con 0x55ff62ecd1b0 2024-09-21T05:51:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c9602d740 10 client.0 _ll_register_callbacks cb 0x7f9c7800b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d855640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39338/0 (socket says 172.21.15.28:39338) 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 -- 172.21.15.28:0/1704936740 learned_addr learned my addr 172.21.15.28:0/1704936740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff632ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 -- 172.21.15.28:0/1704936740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 msgr2=0x55ff632ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 -- 172.21.15.28:0/1704936740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 msgr2=0x55ff632ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff632ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d054640 1 -- 172.21.15.28:0/1704936740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff6332cec0 con 0x55ff62eb49f0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.584+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c8d054640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c840096c0 tx=0x7f9c84032e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697987e7adae73da server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c717fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84035030 con 0x55ff62eb49f0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f9c84005320 con 0x55ff62eb49f0 2024-09-21T05:51:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c840314a0 con 0x55ff62eb49f0 2024-09-21T05:51:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f9c84005520 con 0x55ff62eb49f0 2024-09-21T05:51:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c9602d740 10 client.26261 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ff6332e070 con 0x55ff62eb49f0 2024-09-21T05:51:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.585+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c840314a0 con 0x55ff62eb49f0 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f9c8405e030 con 0x55ff62eb49f0 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 10 client.26261 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 10 client.26261 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 20 client.26261 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62ef6070 0x55ff63330350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 4 client.26261 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff6329b2e0 con 0x55ff62ef6070 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.586+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62ef6070 0x55ff63330350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.587+0000 7f9c8d855640 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62ef6070 0x55ff63330350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e7410fdfa3f1b473 server_cookie=80cc53b849e9aad8 in_seq=0 out_seq=0 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.587+0000 7f9c717fa640 10 client.26261 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.587+0000 7f9c717fa640 1 -- 172.21.15.28:0/1704936740 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff6329b2e0 con 0x55ff62ef6070 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.587+0000 7f9c717fa640 10 client.26261 handle_command_reply: tid=0 2024-09-21T05:51:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.587+0000 7f9c9602d740 1 client.26261 shutdown 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 msgr2=0x55ff632acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c840096c0 tx=0x7f9c84032e30 comp rx=0 tx=0).stop 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 shutdown_connections 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ff62ef6070 0x55ff63330350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c717fa640 0 client.26261 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff62ed5390 0x55ff632ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ff62ecd1b0 0x55ff632ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 --2- 172.21.15.28:0/1704936740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff62eb49f0 0x55ff632acc20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.588+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 >> 172.21.15.28:0/1704936740 conn(0x55ff62eea460 msgr2=0x55ff632a6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.589+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 shutdown_connections 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.589+0000 7f9c9602d740 1 -- 172.21.15.28:0/1704936740 wait complete. 2024-09-21T05:51:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.589+0000 7f9c9602d740 20 client.26261 trim_cache size 0 max 16384 2024-09-21T05:51:37.594 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:51:37.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 msgr2=0x7f9c740b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 0x7f9c740b8c90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9c800e59c0 tx=0x7f9c800a2320 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 msgr2=0x7f9c8819fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000eb10 tx=0x7f9c8000c2d0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 shutdown_connections 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c88105230 0x7f9c8819fd10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9c740b6780 0x7f9c740b8c90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881048e0 0x7f9c8819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 --2- 172.21.15.28:0/2309177848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c88102430 0x7f9c8819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.590+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 >> 172.21.15.28:0/2309177848 conn(0x7f9c88077800 msgr2=0x7f9c88105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.591+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 shutdown_connections 2024-09-21T05:51:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:37.591+0000 7f9c8fab9640 1 -- 172.21.15.28:0/2309177848 wait complete. 2024-09-21T05:51:38.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:38 smithi028 ceph-mon[31227]: pgmap v463: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 7.14k op/s 2024-09-21T05:51:38.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:38 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:38 smithi129 ceph-mon[34885]: pgmap v463: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 7.14k op/s 2024-09-21T05:51:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:38 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:38.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:51:38.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:38 smithi181 ceph-mon[34819]: pgmap v463: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 7.14k op/s 2024-09-21T05:51:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:38 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.547+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/3168756308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db810a9e0 msgr2=0x7f8db810adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.547+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/3168756308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db810a9e0 0x7f8db810adf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009ca0 tx=0x7f8db002a8d0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.549+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/3168756308 shutdown_connections 2024-09-21T05:51:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.549+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/3168756308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db811cc30 0x7f8db811f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.549+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/3168756308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db810a9e0 0x7f8db810adf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.549+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/3168756308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8111f40 0x7f8db810a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.549+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/3168756308 >> 172.21.15.28:0/3168756308 conn(0x7f8db8070ec0 msgr2=0x7f8db80712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.554+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/3168756308 shutdown_connections 2024-09-21T05:51:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.554+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/3168756308 wait complete. 2024-09-21T05:51:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.555+0000 7f8dbcdbd640 1 Processor -- start 2024-09-21T05:51:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.558+0000 7f8dbcdbd640 1 -- start start 2024-09-21T05:51:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 0x7f8db811a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db811cc30 0x7f8db8113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8122960 con 0x7f8db811cc30 2024-09-21T05:51:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81227e0 con 0x7f8db810a9e0 2024-09-21T05:51:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.559+0000 7f8dbcdbd640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8122ae0 con 0x7f8db8111f40 2024-09-21T05:51:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.560+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.562+0000 7f8db659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 0x7f8db811a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.567+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57300/0 (socket says 172.21.15.28:57300) 2024-09-21T05:51:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.567+0000 7f8db5d9b640 1 -- 172.21.15.28:0/2092075973 learned_addr learned my addr 172.21.15.28:0/2092075973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 -- 172.21.15.28:0/2092075973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 msgr2=0x7f8db8113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 -- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db811cc30 msgr2=0x7f8db8113d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db811cc30 0x7f8db8113d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 -- 172.21.15.28:0/2092075973 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db8114590 con 0x7f8db810a9e0 2024-09-21T05:51:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.569+0000 7f8db659c640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 0x7f8db811a490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009fd0 tx=0x7f8dac00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.570+0000 7f8da77fe640 1 -- 172.21.15.28:0/2092075973 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac015020 con 0x7f8db810a9e0 2024-09-21T05:51:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.570+0000 7f8da77fe640 1 -- 172.21.15.28:0/2092075973 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dac011ce0 con 0x7f8db810a9e0 2024-09-21T05:51:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.571+0000 7f8da77fe640 1 -- 172.21.15.28:0/2092075973 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac010e60 con 0x7f8db810a9e0 2024-09-21T05:51:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.573+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db8119220 con 0x7f8db810a9e0 2024-09-21T05:51:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.573+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db8119680 con 0x7f8db810a9e0 2024-09-21T05:51:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.577+0000 7f8dc3331740 1 Processor -- start 2024-09-21T05:51:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.580+0000 7f8da77fe640 1 -- 172.21.15.28:0/2092075973 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8dac018020 con 0x7f8db810a9e0 2024-09-21T05:51:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.582+0000 7f8da77fe640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 0x7f8da00b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.582+0000 7f8da77fe640 1 -- 172.21.15.28:0/2092075973 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8dac0e6c40 con 0x7f8db810a9e0 2024-09-21T05:51:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.583+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 0x7f8da00b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.584+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 0x7f8da00b8c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8da800a860 tx=0x7f8da8005be0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 -- start start 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079e9b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562079e11ea0 con 0x562079ad3390 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562079e012d0 con 0x562079af4070 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.588+0000 7f8dc3331740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562079e050e0 con 0x562079acb1b0 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.589+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.589+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.589+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57306/0 (socket says 172.21.15.28:57306) 2024-09-21T05:51:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.589+0000 7f8db5d9b640 1 -- 172.21.15.28:0/3634917349 learned_addr learned my addr 172.21.15.28:0/3634917349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 -- 172.21.15.28:0/3634917349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 msgr2=0x562079e9b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 -- 172.21.15.28:0/3634917349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 msgr2=0x562079e9b2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:51:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079e9b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 -- 172.21.15.28:0/3634917349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562079b5f650 con 0x562079ad3390 2024-09-21T05:51:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:51:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.590+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9bd40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8da800a640 tx=0x7f8da800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa536f1f0e07c89 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.593+0000 7f8da4ff9640 1 -- 172.21.15.28:0/3634917349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8020070 con 0x562079ad3390 2024-09-21T05:51:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.593+0000 7f8da4ff9640 1 -- 172.21.15.28:0/3634917349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8016070 con 0x562079ad3390 2024-09-21T05:51:39.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.593+0000 7f8da4ff9640 1 -- 172.21.15.28:0/3634917349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da801c3f0 con 0x562079ad3390 2024-09-21T05:51:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.596+0000 7f8dc3331740 1 -- 172.21.15.28:0/3634917349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 msgr2=0x562079e9bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.596+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9bd40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8da800a640 tx=0x7f8da800f000 comp rx=0 tx=0).stop 2024-09-21T05:51:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.599+0000 7f8dc3331740 1 -- 172.21.15.28:0/3634917349 shutdown_connections 2024-09-21T05:51:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.599+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9bd40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.599+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.599+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3634917349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079e9b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.599+0000 7f8dc3331740 1 -- 172.21.15.28:0/3634917349 >> 172.21.15.28:0/3634917349 conn(0x562079ae8460 msgr2=0x562079e991b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.606+0000 7f8dc3331740 1 -- 172.21.15.28:0/3634917349 shutdown_connections 2024-09-21T05:51:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.607+0000 7f8dc3331740 1 -- 172.21.15.28:0/3634917349 wait complete. 2024-09-21T05:51:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.607+0000 7f8dc3331740 1 -- start start 2024-09-21T05:51:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.607+0000 7f8dc3331740 1 Processor -- start 2024-09-21T05:51:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079acb1b0 0x562079ea6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079ea7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562079c62700 con 0x562079ad3390 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562079d9c160 con 0x562079acb1b0 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562079ea9aa0 con 0x562079af4070 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8dc3331740 10 client.0 _ll_register_callbacks cb 0x562079db59e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.609+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57308/0 (socket says 172.21.15.28:57308) 2024-09-21T05:51:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db6d9d640 1 -- 172.21.15.28:0/2860024093 learned_addr learned my addr 172.21.15.28:0/2860024093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079ea7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db6d9d640 1 -- 172.21.15.28:0/2860024093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079acb1b0 msgr2=0x562079ea6ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079acb1b0 0x562079ea6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db6d9d640 1 -- 172.21.15.28:0/2860024093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 msgr2=0x562079ea7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.610+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079ea7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.611+0000 7f8db6d9d640 1 -- 172.21.15.28:0/2860024093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562079dc9c90 con 0x562079af4070 2024-09-21T05:51:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.614+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8da8003c00 tx=0x7f8da8003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957f685fe93fb0a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.614+0000 7f8d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.614+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8020040 con 0x562079af4070 2024-09-21T05:51:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.615+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da8016070 con 0x562079af4070 2024-09-21T05:51:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.615+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8da8004430 con 0x562079af4070 2024-09-21T05:51:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.618+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f8da800fe20 con 0x562079af4070 2024-09-21T05:51:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.618+0000 7f8dc3331740 10 client.26287 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.618+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562079eb5900 con 0x562079af4070 2024-09-21T05:51:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.619+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8016070 con 0x562079af4070 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.620+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f8da802b070 con 0x562079af4070 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 10 client.26287 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 10 client.26287 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 20 client.26287 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079ab29f0 0x562079eb8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 4 client.26287 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.621+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562079e13df0 con 0x562079ab29f0 2024-09-21T05:51:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.624+0000 7f8db659c640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079ab29f0 0x562079eb8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.627+0000 7f8db659c640 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079ab29f0 0x562079eb8d20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2f20b619dca4a528 server_cookie=451b937d96732843 in_seq=0 out_seq=0 2024-09-21T05:51:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.627+0000 7f8d9dffb640 10 client.26287 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:51:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.628+0000 7f8d9dffb640 1 -- 172.21.15.28:0/2860024093 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562079e13df0 con 0x562079ab29f0 2024-09-21T05:51:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.628+0000 7f8d9dffb640 10 client.26287 handle_command_reply: tid=0 2024-09-21T05:51:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.629+0000 7f8dc3331740 1 client.26287 shutdown 2024-09-21T05:51:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.630+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 msgr2=0x562079eb2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.630+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8da8003c00 tx=0x7f8da8003820 comp rx=0 tx=0).stop 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 shutdown_connections 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079ab29f0 0x562079eb8d20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eb2d90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079ea7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 --2- 172.21.15.28:0/2860024093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079acb1b0 0x562079ea6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 >> 172.21.15.28:0/2860024093 conn(0x562079ae8460 msgr2=0x562079e9e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8d9dffb640 0 client.26287 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 shutdown_connections 2024-09-21T05:51:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.634+0000 7f8dc3331740 1 -- 172.21.15.28:0/2860024093 wait complete. 2024-09-21T05:51:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.635+0000 7f8dc3331740 20 client.26287 trim_cache size 0 max 16384 2024-09-21T05:51:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.669+0000 7f8dc3331740 1 Processor -- start 2024-09-21T05:51:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.669+0000 7f8dc3331740 1 -- start start 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.670+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.670+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.670+0000 7f8db659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.670+0000 7f8db659c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57316/0 (socket says 172.21.15.28:57316) 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8db659c640 1 -- 172.21.15.28:0/3647174920 learned_addr learned my addr 172.21.15.28:0/3647174920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079eb2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c6b70 con 0x562079ad3390 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c8710 con 0x562079af4070 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c7240 con 0x562079acb1b0 2024-09-21T05:51:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.671+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079eb2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.674+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.675+0000 7f8db659c640 1 -- 172.21.15.28:0/3647174920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 msgr2=0x562079eb2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.675+0000 7f8db659c640 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079eb2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.675+0000 7f8db659c640 1 -- 172.21.15.28:0/3647174920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 msgr2=0x562079e9ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.675+0000 7f8db659c640 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.675+0000 7f8db659c640 1 -- 172.21.15.28:0/3647174920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562079ea7d70 con 0x562079acb1b0 2024-09-21T05:51:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.683+0000 7f8db659c640 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0ac9e0 tx=0x7f8dac0ac2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9207fee64dc20377 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.684+0000 7f8d9ffff640 1 -- 172.21.15.28:0/3647174920 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0b1530 con 0x562079acb1b0 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.684+0000 7f8d9ffff640 1 -- 172.21.15.28:0/3647174920 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dac0b6470 con 0x562079acb1b0 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.684+0000 7f8d9ffff640 1 -- 172.21.15.28:0/3647174920 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0abcb0 con 0x562079acb1b0 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 msgr2=0x562079e9ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0ac9e0 tx=0x7f8dac0ac2e0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 shutdown_connections 2024-09-21T05:51:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079af4070 0x562079eb2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079ad3390 0x562079e9ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 --2- 172.21.15.28:0/3647174920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079acb1b0 0x562079e9ed80 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0ac9e0 tx=0x7f8dac0ac2e0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.687+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 >> 172.21.15.28:0/3647174920 conn(0x562079ae8460 msgr2=0x562079e9c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.690+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 shutdown_connections 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.690+0000 7f8dc3331740 1 -- 172.21.15.28:0/3647174920 wait complete. 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.691+0000 7f8dc3331740 1 -- start start 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.691+0000 7f8dc3331740 1 Processor -- start 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.692+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 0x562079eaaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.692+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.692+0000 7f8dc3331740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 0x562079f225b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.692+0000 7f8dc3331740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c8040 con 0x562079a87da0 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8dc3331740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c7940 con 0x562079ad3390 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8dc3331740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81c4b50 con 0x562079af4070 2024-09-21T05:51:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8dc3331740 10 client.0 _ll_register_callbacks cb 0x7f8da800c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8db5d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eab470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57320/0 (socket says 172.21.15.28:57320) 2024-09-21T05:51:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8db5d9b640 1 -- 172.21.15.28:0/465114886 learned_addr learned my addr 172.21.15.28:0/465114886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.693+0000 7f8db659c640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 0x562079eaaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 0x562079f225b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 -- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 msgr2=0x562079eab470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eab470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 -- 172.21.15.28:0/465114886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 msgr2=0x562079f225b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 0x562079f225b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 -- 172.21.15.28:0/465114886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562079eabc60 con 0x562079ad3390 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db6d9d640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 0x562079f225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8db659c640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 0x562079eaaf30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00bd70 tx=0x7f8dac0e4320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c7fc886c0b3f2a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.695+0000 7f8d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.696+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0e45d0 con 0x562079ad3390 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.696+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dac0b6470 con 0x562079ad3390 2024-09-21T05:51:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.696+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8dac0e4770 con 0x562079ad3390 2024-09-21T05:51:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.696+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f8dac011830 con 0x562079ad3390 2024-09-21T05:51:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.701+0000 7f8dc3331740 10 client.26270 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.701+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562079f23aa0 con 0x562079ad3390 2024-09-21T05:51:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dac0cb020 con 0x562079ad3390 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f8dac022060 con 0x562079ad3390 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 10 client.26270 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 10 client.26270 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 20 client.26270 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079f24120 0x562079f26540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 4 client.26270 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.706+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8dac0b6470 con 0x562079f24120 2024-09-21T05:51:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.707+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079f24120 0x562079f26540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.711+0000 7f8db5d9b640 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079f24120 0x562079f26540 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7db9672f3766eb70 server_cookie=8fd60a6e1b37ace1 in_seq=0 out_seq=0 2024-09-21T05:51:39.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.712+0000 7f8d9dffb640 10 client.26270 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:51:39.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.712+0000 7f8d9dffb640 1 -- 172.21.15.28:0/465114886 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8dac0b6470 con 0x562079f24120 2024-09-21T05:51:39.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.712+0000 7f8d9dffb640 10 client.26270 handle_command_reply: tid=0 2024-09-21T05:51:39.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.712+0000 7f8dc3331740 1 client.26270 shutdown 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 msgr2=0x562079eaaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 0x562079eaaf30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00bd70 tx=0x7f8dac0e4320 comp rx=0 tx=0).stop 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 shutdown_connections 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x562079f24120 0x562079f26540 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562079a87da0 0x562079f225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8d9dffb640 0 client.26270 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562079af4070 0x562079eab470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 --2- 172.21.15.28:0/465114886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562079ad3390 0x562079eaaf30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 >> 172.21.15.28:0/465114886 conn(0x562079ae8460 msgr2=0x562079e9ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 shutdown_connections 2024-09-21T05:51:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 1 -- 172.21.15.28:0/465114886 wait complete. 2024-09-21T05:51:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.713+0000 7f8dc3331740 20 client.26270 trim_cache size 0 max 16384 2024-09-21T05:51:39.717 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:51:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.714+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 msgr2=0x7f8da00b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.714+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 0x7f8da00b8c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8da800a860 tx=0x7f8da8005be0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.714+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 msgr2=0x7f8db811a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.714+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 0x7f8db811a490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009fd0 tx=0x7f8dac00b870 comp rx=0 tx=0).stop 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 shutdown_connections 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8da00b6730 0x7f8da00b8c40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db811cc30 0x7f8db8113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8db8111f40 0x7f8db8113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 --2- 172.21.15.28:0/2092075973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8db810a9e0 0x7f8db811a490 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.716+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 >> 172.21.15.28:0/2092075973 conn(0x7f8db8070ec0 msgr2=0x7f8db811e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.718+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 shutdown_connections 2024-09-21T05:51:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:39.718+0000 7f8dbcdbd640 1 -- 172.21.15.28:0/2092075973 wait complete. 2024-09-21T05:51:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.733+0000 7fb17b59e640 1 -- 172.21.15.28:0/128803393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10bd60 msgr2=0x7fb17c10c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.733+0000 7fb17b59e640 1 --2- 172.21.15.28:0/128803393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10bd60 0x7fb17c10c170 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009900 tx=0x7fb16c02a440 comp rx=0 tx=0).stop 2024-09-21T05:51:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 -- 172.21.15.28:0/128803393 shutdown_connections 2024-09-21T05:51:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 --2- 172.21.15.28:0/128803393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10b190 0x7fb17c10b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 --2- 172.21.15.28:0/128803393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10c740 0x7fb17c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 --2- 172.21.15.28:0/128803393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10bd60 0x7fb17c10c170 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 -- 172.21.15.28:0/128803393 >> 172.21.15.28:0/128803393 conn(0x7fb17c071050 msgr2=0x7fb17c073b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.734+0000 7fb17b59e640 1 -- 172.21.15.28:0/128803393 shutdown_connections 2024-09-21T05:51:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.737+0000 7fb17b59e640 1 -- 172.21.15.28:0/128803393 wait complete. 2024-09-21T05:51:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.738+0000 7fb17b59e640 1 Processor -- start 2024-09-21T05:51:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.738+0000 7fb17b59e640 1 -- start start 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10b190 0x7fb17c1a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10c740 0x7fb17c1aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c11e2f0 con 0x7fb17c10b190 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c11e170 con 0x7fb17c10c740 2024-09-21T05:51:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17b59e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c11e470 con 0x7fb17c10bd60 2024-09-21T05:51:39.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57324/0 (socket says 172.21.15.28:57324) 2024-09-21T05:51:39.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb179d9b640 1 -- 172.21.15.28:0/1527988879 learned_addr learned my addr 172.21.15.28:0/1527988879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.739+0000 7fb17ad9d640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10c740 0x7fb17c1aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.740+0000 7fb179d9b640 1 -- 172.21.15.28:0/1527988879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10c740 msgr2=0x7fb17c1aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.740+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10c740 0x7fb17c1aa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.740+0000 7fb179d9b640 1 -- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10b190 msgr2=0x7fb17c1a4d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.740+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10b190 0x7fb17c1a4d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.740+0000 7fb179d9b640 1 -- 172.21.15.28:0/1527988879 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c1aac40 con 0x7fb17c10bd60 2024-09-21T05:51:39.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.741+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb17000d740 tx=0x7fb17000dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.744+0000 7fb16b7fe640 1 -- 172.21.15.28:0/1527988879 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170017070 con 0x7fb17c10bd60 2024-09-21T05:51:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.744+0000 7fb16b7fe640 1 -- 172.21.15.28:0/1527988879 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17000de50 con 0x7fb17c10bd60 2024-09-21T05:51:39.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.744+0000 7fb16b7fe640 1 -- 172.21.15.28:0/1527988879 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170003c20 con 0x7fb17c10bd60 2024-09-21T05:51:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.745+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17c1aaf30 con 0x7fb17c10bd60 2024-09-21T05:51:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.745+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17c119170 con 0x7fb17c10bd60 2024-09-21T05:51:39.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.750+0000 7fb1878b3740 1 Processor -- start 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 -- start start 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae514ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5150e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555bae4896f0 con 0x555bae16f1b0 2024-09-21T05:51:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555bae478b20 con 0x555bae1462f0 2024-09-21T05:51:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.751+0000 7fb1878b3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555bae47c930 con 0x555bae14e4d0 2024-09-21T05:51:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57334/0 (socket says 172.21.15.28:57334) 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 -- 172.21.15.28:0/3887569555 learned_addr learned my addr 172.21.15.28:0/3887569555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 -- 172.21.15.28:0/3887569555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 msgr2=0x555bae5150e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 -- 172.21.15.28:0/3887569555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 msgr2=0x555bae514ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae514ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.753+0000 7fb179d9b640 1 -- 172.21.15.28:0/3887569555 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bae43e530 con 0x555bae14e4d0 2024-09-21T05:51:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.754+0000 7fb16b7fe640 1 -- 172.21.15.28:0/1527988879 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb170010460 con 0x7fb17c10bd60 2024-09-21T05:51:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.754+0000 7fb179d9b640 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1700ef1b0 tx=0x7fb1700ef670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9fd18e5a6e95e65 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.758+0000 7fb16b7fe640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 0x7fb1640b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.759+0000 7fb16b7fe640 1 -- 172.21.15.28:0/1527988879 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb1700e6f70 con 0x7fb17c10bd60 2024-09-21T05:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.760+0000 7fb17ad9d640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 0x7fb1640b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.762+0000 7fb168ff9640 1 -- 172.21.15.28:0/3887569555 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700fa070 con 0x555bae14e4d0 2024-09-21T05:51:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.762+0000 7fb168ff9640 1 -- 172.21.15.28:0/3887569555 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1700ac890 con 0x555bae14e4d0 2024-09-21T05:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.763+0000 7fb1878b3740 1 -- 172.21.15.28:0/3887569555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 msgr2=0x555bae515620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.763+0000 7fb1878b3740 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1700ef1b0 tx=0x7fb1700ef670 comp rx=0 tx=0).stop 2024-09-21T05:51:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.763+0000 7fb17ad9d640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 0x7fb1640b8d60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb174009420 tx=0x7fb17400d040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.766+0000 7fb1878b3740 1 -- 172.21.15.28:0/3887569555 shutdown_connections 2024-09-21T05:51:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.766+0000 7fb1878b3740 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae515620 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.766+0000 7fb1878b3740 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5150e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.766+0000 7fb1878b3740 1 --2- 172.21.15.28:0/3887569555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae514ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.766+0000 7fb1878b3740 1 -- 172.21.15.28:0/3887569555 >> 172.21.15.28:0/3887569555 conn(0x555bae163300 msgr2=0x555bae512a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.769+0000 7fb1878b3740 1 -- 172.21.15.28:0/3887569555 shutdown_connections 2024-09-21T05:51:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.770+0000 7fb1878b3740 1 -- 172.21.15.28:0/3887569555 wait complete. 2024-09-21T05:51:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.771+0000 7fb1878b3740 1 -- start start 2024-09-21T05:51:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.771+0000 7fb1878b3740 1 Processor -- start 2024-09-21T05:51:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.771+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae14e4d0 0x555bae520750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae16f1b0 0x555bae52c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555bae3b0ab0 con 0x555bae14e4d0 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555bae41e730 con 0x555bae1462f0 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x555bae523370 con 0x555bae16f1b0 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb1878b3740 10 client.0 _ll_register_callbacks cb 0x555bae4726b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb17a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb17a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39392/0 (socket says 172.21.15.28:39392) 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb17a59c640 1 -- 172.21.15.28:0/1872905730 learned_addr learned my addr 172.21.15.28:0/1872905730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.772+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae14e4d0 0x555bae520750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.773+0000 7fb17a59c640 1 -- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae16f1b0 msgr2=0x555bae52c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.773+0000 7fb17a59c640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae16f1b0 0x555bae52c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.773+0000 7fb17a59c640 1 -- 172.21.15.28:0/1872905730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae14e4d0 msgr2=0x555bae520750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.773+0000 7fb17a59c640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae14e4d0 0x555bae520750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.773+0000 7fb17a59c640 1 -- 172.21.15.28:0/1872905730 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bae1da790 con 0x555bae1462f0 2024-09-21T05:51:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.774+0000 7fb17a59c640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009230 tx=0x7fb16c003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7dfaed8bd24b197 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.776+0000 7fb161ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.778+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c040030 con 0x555bae1462f0 2024-09-21T05:51:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.778+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb16c031790 con 0x555bae1462f0 2024-09-21T05:51:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.779+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1700ac890 con 0x555bae1462f0 2024-09-21T05:51:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.782+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb16c031990 con 0x555bae1462f0 2024-09-21T05:51:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.784+0000 7fb1878b3740 10 client.26276 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.784+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555bae253620 con 0x555bae1462f0 2024-09-21T05:51:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.785+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1700ac890 con 0x555bae1462f0 2024-09-21T05:51:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.785+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fb16c04f420 con 0x555bae1462f0 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 10 client.26276 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 10 client.26276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 20 client.26276 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae12dd80 0x555bae5434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 4 client.26276 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.786+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bae48b640 con 0x555bae12dd80 2024-09-21T05:51:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.787+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae12dd80 0x555bae5434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.793+0000 7fb179d9b640 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae12dd80 0x555bae5434f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83a33fff16f24f48 server_cookie=201f70895fb05ab in_seq=0 out_seq=0 2024-09-21T05:51:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.794+0000 7fb161ffb640 10 client.26276 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:39.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.795+0000 7fb161ffb640 1 -- 172.21.15.28:0/1872905730 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555bae48b640 con 0x555bae12dd80 2024-09-21T05:51:39.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.795+0000 7fb161ffb640 10 client.26276 handle_command_reply: tid=0 2024-09-21T05:51:39.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.795+0000 7fb1878b3740 1 client.26276 shutdown 2024-09-21T05:51:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.796+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 msgr2=0x555bae520210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.796+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009230 tx=0x7fb16c003730 comp rx=0 tx=0).stop 2024-09-21T05:51:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 shutdown_connections 2024-09-21T05:51:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae12dd80 0x555bae5434f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae16f1b0 0x555bae52c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae14e4d0 0x555bae520750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 --2- 172.21.15.28:0/1872905730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae520210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 >> 172.21.15.28:0/1872905730 conn(0x555bae163300 msgr2=0x555bae515bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.800+0000 7fb161ffb640 0 client.26276 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.802+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 shutdown_connections 2024-09-21T05:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.805+0000 7fb1878b3740 1 -- 172.21.15.28:0/1872905730 wait complete. 2024-09-21T05:51:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.808+0000 7fb1878b3740 20 client.26276 trim_cache size 0 max 16384 2024-09-21T05:51:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.843+0000 7fb1878b3740 1 Processor -- start 2024-09-21T05:51:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.843+0000 7fb1878b3740 1 -- start start 2024-09-21T05:51:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.844+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.844+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae51a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.844+0000 7fb17a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.844+0000 7fb17a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39398/0 (socket says 172.21.15.28:39398) 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.844+0000 7fb17a59c640 1 -- 172.21.15.28:0/51415003 learned_addr learned my addr 172.21.15.28:0/51415003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.845+0000 7fb1878b3740 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae522740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.845+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00e910 con 0x555bae16f1b0 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.845+0000 7fb17ad9d640 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae522740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.845+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c0104b0 con 0x555bae1462f0 2024-09-21T05:51:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.845+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00efe0 con 0x555bae14e4d0 2024-09-21T05:51:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.847+0000 7fb17a59c640 1 -- 172.21.15.28:0/51415003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 msgr2=0x555bae51a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.847+0000 7fb17a59c640 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae51a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.847+0000 7fb17a59c640 1 -- 172.21.15.28:0/51415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 msgr2=0x555bae522740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.847+0000 7fb17a59c640 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae522740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.847+0000 7fb17a59c640 1 -- 172.21.15.28:0/51415003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bae517310 con 0x555bae1462f0 2024-09-21T05:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.848+0000 7fb17a59c640 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb16c003820 tx=0x7fb16c02a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543b207f9aba005a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.848+0000 7fb163fff640 1 -- 172.21.15.28:0/51415003 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c000e60 con 0x555bae1462f0 2024-09-21T05:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.848+0000 7fb163fff640 1 -- 172.21.15.28:0/51415003 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb16c009520 con 0x555bae1462f0 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 msgr2=0x555bae5206f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb16c003820 tx=0x7fb16c02a440 comp rx=0 tx=0).stop 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 shutdown_connections 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae522740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae51a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 --2- 172.21.15.28:0/51415003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae5206f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.850+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 >> 172.21.15.28:0/51415003 conn(0x555bae163300 msgr2=0x555bae519c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.853+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 shutdown_connections 2024-09-21T05:51:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.853+0000 7fb1878b3740 1 -- 172.21.15.28:0/51415003 wait complete. 2024-09-21T05:51:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.854+0000 7fb1878b3740 1 -- start start 2024-09-21T05:51:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.854+0000 7fb1878b3740 1 Processor -- start 2024-09-21T05:51:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae525790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae526210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00fde0 con 0x555bae16f1b0 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00f6e0 con 0x555bae1462f0 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00c8f0 con 0x555bae14e4d0 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.855+0000 7fb1878b3740 10 client.0 _ll_register_callbacks cb 0x7fb16c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57358/0 (socket says 172.21.15.28:57358) 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 -- 172.21.15.28:0/850968232 learned_addr learned my addr 172.21.15.28:0/850968232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb17ad9d640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae526210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb17a59c640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae525790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 -- 172.21.15.28:0/850968232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 msgr2=0x555bae525790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae525790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 -- 172.21.15.28:0/850968232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 msgr2=0x555bae526210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae526210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.856+0000 7fb179d9b640 1 -- 172.21.15.28:0/850968232 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bae5ac070 con 0x555bae14e4d0 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.857+0000 7fb179d9b640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a9ff0 tx=0x7fb1700a8de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3ebdd55b497a97 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.857+0000 7fb161ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.859+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170020040 con 0x555bae14e4d0 2024-09-21T05:51:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.859+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb1700e4900 con 0x555bae14e4d0 2024-09-21T05:51:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.861+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555bae4870f0 con 0x555bae14e4d0 2024-09-21T05:51:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.864+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb1700ac4f0 con 0x555bae14e4d0 2024-09-21T05:51:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.864+0000 7fb1878b3740 10 client.26305 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.864+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555bae5ad370 con 0x555bae14e4d0 2024-09-21T05:51:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.867+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555bae4870f0 con 0x555bae14e4d0 2024-09-21T05:51:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.867+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fb1700b4be0 con 0x555bae14e4d0 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 10 client.26305 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 20 client.26305 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae1031c0 0x555bae5af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 4 client.26305 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:51:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.870+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555bae484520 con 0x555bae1031c0 2024-09-21T05:51:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.871+0000 7fb17a59c640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae1031c0 0x555bae5af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.873+0000 7fb17a59c640 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae1031c0 0x555bae5af670 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c2d466c7020917a server_cookie=fe97cd6546002720 in_seq=0 out_seq=0 2024-09-21T05:51:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:39.873+0000 7fb161ffb640 10 client.26305 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb161ffb640 1 -- 172.21.15.28:0/850968232 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555bae484520 con 0x555bae1031c0 2024-09-21T05:51:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb161ffb640 10 client.26305 handle_command_reply: tid=0 2024-09-21T05:51:40.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 client.26305 shutdown 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 msgr2=0x555bae525cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a9ff0 tx=0x7fb1700a8de0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 shutdown_connections 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x555bae1031c0 0x555bae5af670 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bae16f1b0 0x555bae526210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb161ffb640 0 client.26305 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bae14e4d0 0x555bae525cd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 --2- 172.21.15.28:0/850968232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555bae1462f0 0x555bae525790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 >> 172.21.15.28:0/850968232 conn(0x555bae163300 msgr2=0x555bae51f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 shutdown_connections 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 1 -- 172.21.15.28:0/850968232 wait complete. 2024-09-21T05:51:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.162+0000 7fb1878b3740 20 client.26305 trim_cache size 0 max 16384 2024-09-21T05:51:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:51:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:51:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:51:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 msgr2=0x7fb1640b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 0x7fb1640b8d60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb174009420 tx=0x7fb17400d040 comp rx=0 tx=0).stop 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 msgr2=0x7fb17c1a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb17000d740 tx=0x7fb17000dc00 comp rx=0 tx=0).stop 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 shutdown_connections 2024-09-21T05:51:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17c10c740 0x7fb17c1aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb17c10bd60 0x7fb17c1a52b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1640b6850 0x7fb1640b8d60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 --2- 172.21.15.28:0/1527988879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb17c10b190 0x7fb17c1a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 >> 172.21.15.28:0/1527988879 conn(0x7fb17c071050 msgr2=0x7fb17c116fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 shutdown_connections 2024-09-21T05:51:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:40.163+0000 7fb17b59e640 1 -- 172.21.15.28:0/1527988879 wait complete. 2024-09-21T05:51:40.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:40 smithi129 ceph-mon[34885]: pgmap v464: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 MiB/s wr, 7.48k op/s 2024-09-21T05:51:40.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:40 smithi181 ceph-mon[34819]: pgmap v464: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 MiB/s wr, 7.48k op/s 2024-09-21T05:51:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:40 smithi028 ceph-mon[31227]: pgmap v464: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 MiB/s wr, 7.48k op/s 2024-09-21T05:51:40.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:51:41.288 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 -- 172.21.15.28:0/4246134908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7901081b0 msgr2=0x7fe7901085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 --2- 172.21.15.28:0/4246134908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7901081b0 0x7fe7901085c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe780009900 tx=0x7fe78002a3f0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 -- 172.21.15.28:0/4246134908 shutdown_connections 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 --2- 172.21.15.28:0/4246134908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe790108b00 0x7fe79010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 --2- 172.21.15.28:0/4246134908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7901081b0 0x7fe7901085c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 --2- 172.21.15.28:0/4246134908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790102870 0x7fe790102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.912+0000 7fe79480a640 1 -- 172.21.15.28:0/4246134908 >> 172.21.15.28:0/4246134908 conn(0x7fe7900fe2c0 msgr2=0x7fe790100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.913+0000 7fe79480a640 1 -- 172.21.15.28:0/4246134908 shutdown_connections 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.913+0000 7fe79480a640 1 -- 172.21.15.28:0/4246134908 wait complete. 2024-09-21T05:51:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 Processor -- start 2024-09-21T05:51:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 -- start start 2024-09-21T05:51:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7901081b0 0x7fe79010e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 0x7fe7901a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe790110fd0 con 0x7fe790102870 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe790110e50 con 0x7fe790108b00 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.914+0000 7fe79480a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe790111150 con 0x7fe7901081b0 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33216/0 (socket says 172.21.15.28:33216) 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 -- 172.21.15.28:0/4147347609 learned_addr learned my addr 172.21.15.28:0/4147347609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78f59e640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 0x7fe7901a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe787fff640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7901081b0 0x7fe79010e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 -- 172.21.15.28:0/4147347609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7901081b0 msgr2=0x7fe79010e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7901081b0 0x7fe79010e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 -- 172.21.15.28:0/4147347609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 msgr2=0x7fe7901a5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 0x7fe7901a5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78ed9d640 1 -- 172.21.15.28:0/4147347609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7901a5770 con 0x7fe790102870 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.915+0000 7fe78f59e640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 0x7fe7901a5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe7780097d0 tx=0x7fe778009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe78cd99640 1 -- 172.21.15.28:0/4147347609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77800eab0 con 0x7fe790102870 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe78cd99640 1 -- 172.21.15.28:0/4147347609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe778019070 con 0x7fe790102870 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7901a5a60 con 0x7fe790102870 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe78cd99640 1 -- 172.21.15.28:0/4147347609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778013360 con 0x7fe790102870 2024-09-21T05:51:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.916+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7901ad290 con 0x7fe790102870 2024-09-21T05:51:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 Processor -- start 2024-09-21T05:51:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 -- start start 2024-09-21T05:51:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c83da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d89bf9ea0 con 0x561d898b31b0 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d89be92d0 con 0x561d898dc070 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.920+0000 7fe79b57f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d89bed0e0 con 0x561d898bb390 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe78f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe78f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39420/0 (socket says 172.21.15.28:39420) 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe78f59e640 1 -- 172.21.15.28:0/2099515877 learned_addr learned my addr 172.21.15.28:0/2099515877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe787fff640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe78cd99640 1 -- 172.21.15.28:0/4147347609 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe778013070 con 0x7fe790102870 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.921+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c83da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 -- 172.21.15.28:0/2099515877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 msgr2=0x561d89c83860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 -- 172.21.15.28:0/2099515877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 msgr2=0x561d89c83da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c83da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 -- 172.21.15.28:0/2099515877 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d89947650 con 0x561d898bb390 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe78f59e640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe787fff640 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c842e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe78002a910 tx=0x7fe780031530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b13610b5643077ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe7857fa640 1 -- 172.21.15.28:0/2099515877 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780035020 con 0x561d898bb390 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe7857fa640 1 -- 172.21.15.28:0/2099515877 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780041070 con 0x561d898bb390 2024-09-21T05:51:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.922+0000 7fe7857fa640 1 -- 172.21.15.28:0/2099515877 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780004bf0 con 0x561d898bb390 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.923+0000 7fe78cd99640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 0x7fe77c0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.924+0000 7fe78f59e640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 0x7fe77c0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.924+0000 7fe78cd99640 1 -- 172.21.15.28:0/4147347609 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe7780e7130 con 0x7fe790102870 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.924+0000 7fe79b57f740 1 -- 172.21.15.28:0/2099515877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 msgr2=0x561d89c842e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.924+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c842e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe78002a910 tx=0x7fe780031530 comp rx=0 tx=0).stop 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.924+0000 7fe78f59e640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 0x7fe77c0b8c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe788010b30 tx=0x7fe7880096a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 -- 172.21.15.28:0/2099515877 shutdown_connections 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c842e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c83da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2099515877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c83860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 -- 172.21.15.28:0/2099515877 >> 172.21.15.28:0/2099515877 conn(0x561d898d0460 msgr2=0x561d89c81750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 -- 172.21.15.28:0/2099515877 shutdown_connections 2024-09-21T05:51:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.925+0000 7fe79b57f740 1 -- 172.21.15.28:0/2099515877 wait complete. 2024-09-21T05:51:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.926+0000 7fe79b57f740 1 -- start start 2024-09-21T05:51:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.926+0000 7fe79b57f740 1 Processor -- start 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c8f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c9b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d89a4a700 con 0x561d898b31b0 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d89b84160 con 0x561d898dc070 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561d89c91fe0 con 0x561d898bb390 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe79b57f740 10 client.0 _ll_register_callbacks cb 0x561d89b9d9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33242/0 (socket says 172.21.15.28:33242) 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe78ed9d640 1 -- 172.21.15.28:0/2648999138 learned_addr learned my addr 172.21.15.28:0/2648999138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe78f59e640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c9b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe78ed9d640 1 -- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 msgr2=0x561d89c8f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.927+0000 7fe787fff640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c8f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c8f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78ed9d640 1 -- 172.21.15.28:0/2648999138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 msgr2=0x561d89c9b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c9b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78ed9d640 1 -- 172.21.15.28:0/2648999138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d89bb1c90 con 0x561d898b31b0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78f59e640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c9b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe7780aa1b0 tx=0x7fe7780a8ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db37c86ae9cdd35 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe7767fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7780ac6b0 con 0x561d898b31b0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe7780e90d0 con 0x561d898b31b0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.928+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780041070 con 0x561d898b31b0 2024-09-21T05:51:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe7780aca50 con 0x561d898b31b0 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 10 client.16713 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561d89c9dcb0 con 0x561d898b31b0 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780041070 con 0x561d898b31b0 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fe7780abc10 con 0x561d898b31b0 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 10 client.16713 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 10 client.16713 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 20 client.16713 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8989a9f0 0x561d89ca1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 4 client.16713 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.929+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d89bfbdf0 con 0x561d8989a9f0 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.930+0000 7fe787fff640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8989a9f0 0x561d89ca1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.931+0000 7fe7767fc640 10 client.16713 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.931+0000 7fe787fff640 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8989a9f0 0x561d89ca1690 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b54265635f218a9 server_cookie=265522032504087c in_seq=0 out_seq=0 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.932+0000 7fe7767fc640 1 -- 172.21.15.28:0/2648999138 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561d89bfbdf0 con 0x561d8989a9f0 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.932+0000 7fe7767fc640 10 client.16713 handle_command_reply: tid=0 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.932+0000 7fe79b57f740 1 client.16713 shutdown 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 msgr2=0x561d89c8ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe7780aa1b0 tx=0x7fe7780a8ea0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 shutdown_connections 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8989a9f0 0x561d89ca1690 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89c9b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898bb390 0x561d89c8f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2648999138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8ef40 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 >> 172.21.15.28:0/2648999138 conn(0x561d898d0460 msgr2=0x561d89c86a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe7767fc640 0 client.16713 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 shutdown_connections 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 1 -- 172.21.15.28:0/2648999138 wait complete. 2024-09-21T05:51:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.933+0000 7fe79b57f740 20 client.16713 trim_cache size 0 max 16384 2024-09-21T05:51:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.957+0000 7fe79b57f740 1 Processor -- start 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.957+0000 7fe79b57f740 1 -- start start 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 0x561d89c9b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898dc070 0x561d89c80d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b5b90 con 0x561d898b31b0 2024-09-21T05:51:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b7730 con 0x561d898bb390 2024-09-21T05:51:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe79b57f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b6260 con 0x561d898dc070 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33256/0 (socket says 172.21.15.28:33256) 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe78ed9d640 1 -- 172.21.15.28:0/2278857914 learned_addr learned my addr 172.21.15.28:0/2278857914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe787fff640 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 0x561d89c9b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.958+0000 7fe78f59e640 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898dc070 0x561d89c80d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.959+0000 7fe787fff640 1 -- 172.21.15.28:0/2278857914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898dc070 msgr2=0x561d89c80d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.959+0000 7fe787fff640 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898dc070 0x561d89c80d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.959+0000 7fe787fff640 1 -- 172.21.15.28:0/2278857914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 msgr2=0x561d89c8f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.959+0000 7fe787fff640 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.959+0000 7fe787fff640 1 -- 172.21.15.28:0/2278857914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d89c7cb80 con 0x561d898bb390 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.960+0000 7fe787fff640 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 0x561d89c9b1d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x561d89ca1e30 tx=0x7fe78002a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dceb27bfb1543099 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.960+0000 7fe784ff9640 1 -- 172.21.15.28:0/2278857914 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780053070 con 0x561d898bb390 2024-09-21T05:51:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.961+0000 7fe784ff9640 1 -- 172.21.15.28:0/2278857914 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780041070 con 0x561d898bb390 2024-09-21T05:51:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.961+0000 7fe784ff9640 1 -- 172.21.15.28:0/2278857914 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780035040 con 0x561d898bb390 2024-09-21T05:51:41.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 -- 172.21.15.28:0/2278857914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 msgr2=0x561d89c9b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 0x561d89c9b1d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x561d89ca1e30 tx=0x7fe78002a3f0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 -- 172.21.15.28:0/2278857914 shutdown_connections 2024-09-21T05:51:41.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898dc070 0x561d89c80d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898bb390 0x561d89c9b1d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 --2- 172.21.15.28:0/2278857914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898b31b0 0x561d89c8f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.962+0000 7fe79b57f740 1 -- 172.21.15.28:0/2278857914 >> 172.21.15.28:0/2278857914 conn(0x561d898d0460 msgr2=0x561d89c8ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.963+0000 7fe79b57f740 1 -- 172.21.15.28:0/2278857914 shutdown_connections 2024-09-21T05:51:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.963+0000 7fe79b57f740 1 -- 172.21.15.28:0/2278857914 wait complete. 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.964+0000 7fe79b57f740 1 -- start start 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.964+0000 7fe79b57f740 1 Processor -- start 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.964+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898bb390 0x561d89c93000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89d0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe78ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57390/0 (socket says 172.21.15.28:57390) 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b7060 con 0x561d898bb390 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b6960 con 0x561d898dc070 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe78ed9d640 1 -- 172.21.15.28:0/3792878780 learned_addr learned my addr 172.21.15.28:0/3792878780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901b3b70 con 0x561d898b31b0 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe787fff640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898bb390 0x561d89c93000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.965+0000 7fe79b57f740 10 client.0 _ll_register_callbacks cb 0x7fe7780a9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 -- 172.21.15.28:0/3792878780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 msgr2=0x561d89d0a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89d0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 -- 172.21.15.28:0/3792878780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898bb390 msgr2=0x561d89c93000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898bb390 0x561d89c93000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 -- 172.21.15.28:0/3792878780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d89d0a6f0 con 0x561d898b31b0 2024-09-21T05:51:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe78ed9d640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7780ace80 tx=0x7fe7780a8f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a097c03813451ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.966+0000 7fe7767fc640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.967+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7780b1a20 con 0x561d898b31b0 2024-09-21T05:51:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.967+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fe7780d41f0 con 0x561d898b31b0 2024-09-21T05:51:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.967+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780041070 con 0x561d898b31b0 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.967+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe7780b6960 con 0x561d898b31b0 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 10 client.26320 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561d89d0b960 con 0x561d898b31b0 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7780d43f0 con 0x561d898b31b0 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7fe7780b2bb0 con 0x561d898b31b0 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 10 client.26320 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 10 client.26320 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 20 client.26320 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8986fda0 0x561d89d0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 4 client.26320 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.968+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe780041070 con 0x561d8986fda0 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.969+0000 7fe787fff640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8986fda0 0x561d89d0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.969+0000 7fe7767fc640 10 client.26320 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.969+0000 7fe787fff640 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8986fda0 0x561d89d0df70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6cd724ff8e5ca6b server_cookie=5ed7605e20939eb3 in_seq=0 out_seq=0 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.970+0000 7fe7767fc640 1 -- 172.21.15.28:0/3792878780 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe780041070 con 0x561d8986fda0 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.970+0000 7fe7767fc640 10 client.26320 handle_command_reply: tid=0 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.970+0000 7fe79b57f740 1 client.26320 shutdown 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.970+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 msgr2=0x561d89c92a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.970+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7780ace80 tx=0x7fe7780a8f10 comp rx=0 tx=0).stop 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 shutdown_connections 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x561d8986fda0 0x561d89d0df70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d898dc070 0x561d89d0a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe7767fc640 0 client.26320 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d898bb390 0x561d89c93000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 --2- 172.21.15.28:0/3792878780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561d898b31b0 0x561d89c92a40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 >> 172.21.15.28:0/3792878780 conn(0x561d898d0460 msgr2=0x561d89c84890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.971+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 shutdown_connections 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.972+0000 7fe79b57f740 1 -- 172.21.15.28:0/3792878780 wait complete. 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.972+0000 7fe79b57f740 20 client.26320 trim_cache size 0 max 16384 2024-09-21T05:51:41.976 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 msgr2=0x7fe77c0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 0x7fe77c0b8c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe788010b30 tx=0x7fe7880096a0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 msgr2=0x7fe79010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe7780097d0 tx=0x7fe778009c90 comp rx=0 tx=0).stop 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 shutdown_connections 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe790108b00 0x7fe7901a5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7901081b0 0x7fe79010e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe77c0b6780 0x7fe77c0b8c90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 --2- 172.21.15.28:0/4147347609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe790102870 0x7fe79010e1a0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:41.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.973+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 >> 172.21.15.28:0/4147347609 conn(0x7fe7900fe2c0 msgr2=0x7fe790113740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:41.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.974+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 shutdown_connections 2024-09-21T05:51:41.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:41.974+0000 7fe79480a640 1 -- 172.21.15.28:0/4147347609 wait complete. 2024-09-21T05:51:42.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:42 smithi028 ceph-mon[31227]: pgmap v465: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 6.73k op/s 2024-09-21T05:51:42.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:42 smithi181 ceph-mon[34819]: pgmap v465: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 6.73k op/s 2024-09-21T05:51:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:42 smithi129 ceph-mon[34885]: pgmap v465: 97 pgs: 97 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 MiB/s wr, 6.73k op/s 2024-09-21T05:51:42.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:51:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.913+0000 7f3e677fe640 1 -- 172.21.15.28:0/3730536886 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580055c0 con 0x7f3e6810baf0 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.913+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3730536886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e6810baf0 msgr2=0x7f3e6810bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3730536886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e6810baf0 0x7f3e6810bf00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3e58008570 tx=0x7f3e5800ac00 comp rx=0 tx=0).stop 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3730536886 shutdown_connections 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3730536886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e68110490 0x7f3e6807b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3730536886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810fb40 0x7f3e6810ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3730536886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e6810baf0 0x7f3e6810bf00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3730536886 >> 172.21.15.28:0/3730536886 conn(0x7f3e68070ec0 msgr2=0x7f3e680712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.914+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3730536886 shutdown_connections 2024-09-21T05:51:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.915+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3730536886 wait complete. 2024-09-21T05:51:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.916+0000 7f3e6f0b4640 1 Processor -- start 2024-09-21T05:51:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.917+0000 7f3e6f0b4640 1 -- start start 2024-09-21T05:51:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e6810fb40 0x7f3e68080b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e68110490 0x7f3e680810c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6807ec60 con 0x7f3e6810baf0 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6807eae0 con 0x7f3e68110490 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6f0b4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6807ede0 con 0x7f3e6810fb40 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33276/0 (socket says 172.21.15.28:33276) 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3043518385 learned_addr learned my addr 172.21.15.28:0/3043518385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.918+0000 7f3e6d651640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e68110490 0x7f3e680810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.919+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3043518385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e6810fb40 msgr2=0x7f3e68080b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.919+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e6810fb40 0x7f3e68080b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.919+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3043518385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e68110490 msgr2=0x7f3e680810c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.919+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e68110490 0x7f3e680810c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.919+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3043518385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e681cec30 con 0x7f3e6810baf0 2024-09-21T05:51:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.920+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3e58005ab0 tx=0x7f3e5800a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:43.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.920+0000 7f3e65ffb640 1 -- 172.21.15.28:0/3043518385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58004250 con 0x7f3e6810baf0 2024-09-21T05:51:43.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.920+0000 7f3e65ffb640 1 -- 172.21.15.28:0/3043518385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e580043f0 con 0x7f3e6810baf0 2024-09-21T05:51:43.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.920+0000 7f3e65ffb640 1 -- 172.21.15.28:0/3043518385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58032940 con 0x7f3e6810baf0 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.920+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e681ceec0 con 0x7f3e6810baf0 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.921+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e681cf260 con 0x7f3e6810baf0 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.924+0000 7f3e75628740 1 Processor -- start 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 -- start start 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f89606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f8598390 0x55f0f8961120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0f88d6ea0 con 0x55f0f8598390 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f0f88c62d0 con 0x55f0f85901b0 2024-09-21T05:51:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e75628740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f0f88ca0e0 con 0x55f0f85b9070 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:39480/0 (socket says 172.21.15.28:39480) 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2613682202 learned_addr learned my addr 172.21.15.28:0/2613682202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.925+0000 7f3e6d651640 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f89606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.926+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2613682202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 msgr2=0x55f0f89606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.926+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f89606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.926+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2613682202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f8598390 msgr2=0x55f0f8961120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.926+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f8598390 0x55f0f8961120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.926+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2613682202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f8624650 con 0x55f0f85901b0 2024-09-21T05:51:43.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.928+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e58003c30 tx=0x7f3e58032440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59bb7cdee119981a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:43.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.930+0000 7f3e65ffb640 1 -- 172.21.15.28:0/3043518385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3e58003800 con 0x7f3e6810baf0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.932+0000 7f3e65ffb640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 0x7f3e500b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.932+0000 7f3e6d651640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 0x7f3e500b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.933+0000 7f3e4effd640 1 -- 172.21.15.28:0/2613682202 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e581092d0 con 0x55f0f85901b0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.933+0000 7f3e6d651640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 0x7f3e500b8d10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e680865d0 tx=0x7f3e5c00f040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.933+0000 7f3e65ffb640 1 -- 172.21.15.28:0/3043518385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3e5804d070 con 0x7f3e6810baf0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.933+0000 7f3e4effd640 1 -- 172.21.15.28:0/2613682202 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e58109b10 con 0x55f0f85901b0 2024-09-21T05:51:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.933+0000 7f3e4effd640 1 -- 172.21.15.28:0/2613682202 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5810ad50 con 0x55f0f85901b0 2024-09-21T05:51:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.935+0000 7f3e75628740 1 -- 172.21.15.28:0/2613682202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 msgr2=0x55f0f8960be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.935+0000 7f3e75628740 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e58003c30 tx=0x7f3e58032440 comp rx=0 tx=0).stop 2024-09-21T05:51:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.938+0000 7f3e75628740 1 -- 172.21.15.28:0/2613682202 shutdown_connections 2024-09-21T05:51:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.938+0000 7f3e75628740 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f8598390 0x55f0f8961120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.938+0000 7f3e75628740 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85901b0 0x55f0f8960be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.938+0000 7f3e75628740 1 --2- 172.21.15.28:0/2613682202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f89606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.938+0000 7f3e75628740 1 -- 172.21.15.28:0/2613682202 >> 172.21.15.28:0/2613682202 conn(0x55f0f85ad460 msgr2=0x55f0f895e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.943+0000 7f3e75628740 1 -- 172.21.15.28:0/2613682202 shutdown_connections 2024-09-21T05:51:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.944+0000 7f3e75628740 1 -- 172.21.15.28:0/2613682202 wait complete. 2024-09-21T05:51:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.945+0000 7f3e75628740 1 -- start start 2024-09-21T05:51:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.945+0000 7f3e75628740 1 Processor -- start 2024-09-21T05:51:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.945+0000 7f3e75628740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f896bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f8598390 0x55f0f896c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f8977db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8727700 con 0x55f0f85901b0 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f0f8861160 con 0x55f0f8598390 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f0f88d6ea0 con 0x55f0f85b9070 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e75628740 10 client.0 _ll_register_callbacks cb 0x55f0f887a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e6d651640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f8977db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e6d651640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f8977db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57406/0 (socket says 172.21.15.28:57406) 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.946+0000 7f3e6d651640 1 -- 172.21.15.28:0/3228794439 learned_addr learned my addr 172.21.15.28:0/3228794439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f896bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3228794439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 msgr2=0x55f0f8977db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f8977db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f8598390 msgr2=0x55f0f896c2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f8598390 0x55f0f896c2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.947+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3228794439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f888ec90 con 0x55f0f85901b0 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.948+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f896bd80 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3e580047b0 tx=0x7f3e5810a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e799588ceaa77f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.948+0000 7f3e47fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:43.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.948+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580cea30 con 0x55f0f85901b0 2024-09-21T05:51:43.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.948+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3e580cb350 con 0x55f0f85901b0 2024-09-21T05:51:43.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.949+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e58109b10 con 0x55f0f85901b0 2024-09-21T05:51:43.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.949+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3e580ce850 con 0x55f0f85901b0 2024-09-21T05:51:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.949+0000 7f3e75628740 10 client.16728 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.949+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f0f897a800 con 0x55f0f85901b0 2024-09-21T05:51:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e580cebf0 con 0x55f0f85901b0 2024-09-21T05:51:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f3e58044070 con 0x55f0f85901b0 2024-09-21T05:51:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 10 client.16728 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 10 client.16728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 20 client.16728 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f85779f0 0x55f0f897dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 4 client.16728 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3e58109b10 con 0x55f0f85779f0 2024-09-21T05:51:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.950+0000 7f3e67fff640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f85779f0 0x55f0f897dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.956+0000 7f3e47fff640 10 client.16728 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:51:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.956+0000 7f3e67fff640 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f85779f0 0x55f0f897dc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e7420cdfa5caf80 server_cookie=f84b2d5dbdf607ff in_seq=0 out_seq=0 2024-09-21T05:51:43.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.958+0000 7f3e47fff640 1 -- 172.21.15.28:0/3228794439 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f3e58109b10 con 0x55f0f85779f0 2024-09-21T05:51:43.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.958+0000 7f3e47fff640 10 client.16728 handle_command_reply: tid=0 2024-09-21T05:51:43.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.958+0000 7f3e75628740 1 client.16728 shutdown 2024-09-21T05:51:43.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.968+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 msgr2=0x55f0f896bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:43.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.968+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f896bd80 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3e580047b0 tx=0x7f3e5810a810 comp rx=0 tx=0).stop 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 shutdown_connections 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f85779f0 0x55f0f897dc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f85b9070 0x55f0f8977db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f8598390 0x55f0f896c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 --2- 172.21.15.28:0/3228794439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f896bd80 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 >> 172.21.15.28:0/3228794439 conn(0x55f0f85ad460 msgr2=0x55f0f89616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:43.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.969+0000 7f3e47fff640 0 client.16728 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:51:43.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.971+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 shutdown_connections 2024-09-21T05:51:43.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.971+0000 7f3e75628740 1 -- 172.21.15.28:0/3228794439 wait complete. 2024-09-21T05:51:43.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:43.973+0000 7f3e75628740 20 client.16728 trim_cache size 0 max 16384 2024-09-21T05:51:44.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.001+0000 7f3e75628740 1 Processor -- start 2024-09-21T05:51:44.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.001+0000 7f3e75628740 1 -- start start 2024-09-21T05:51:44.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f8968f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d9950 con 0x55f0f85901b0 2024-09-21T05:51:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d8480 con 0x55f0f85b9070 2024-09-21T05:51:44.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e75628740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d8b80 con 0x55f0f8598390 2024-09-21T05:51:44.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.002+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33306/0 (socket says 172.21.15.28:33306) 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2775496778 learned_addr learned my addr 172.21.15.28:0/2775496778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2775496778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 msgr2=0x55f0f896c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2775496778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 msgr2=0x55f0f8968f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f8968f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 -- 172.21.15.28:0/2775496778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f8963500 con 0x55f0f85901b0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.003+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3e580cede0 tx=0x7f3e58032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515ee5dd150fe2f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.004+0000 7f3e4e7fc640 1 -- 172.21.15.28:0/2775496778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580cea30 con 0x55f0f85901b0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.004+0000 7f3e4e7fc640 1 -- 172.21.15.28:0/2775496778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e580cebf0 con 0x55f0f85901b0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.004+0000 7f3e4e7fc640 1 -- 172.21.15.28:0/2775496778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580cb350 con 0x55f0f85901b0 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.005+0000 7f3e75628740 1 -- 172.21.15.28:0/2775496778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 msgr2=0x55f0f8961d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.005+0000 7f3e75628740 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3e580cede0 tx=0x7f3e58032440 comp rx=0 tx=0).stop 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 -- 172.21.15.28:0/2775496778 shutdown_connections 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f8968f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 --2- 172.21.15.28:0/2775496778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8961d40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 -- 172.21.15.28:0/2775496778 >> 172.21.15.28:0/2775496778 conn(0x55f0f85ad460 msgr2=0x55f0f896bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 -- 172.21.15.28:0/2775496778 shutdown_connections 2024-09-21T05:51:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.006+0000 7f3e75628740 1 -- 172.21.15.28:0/2775496778 wait complete. 2024-09-21T05:51:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.007+0000 7f3e75628740 1 -- start start 2024-09-21T05:51:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.008+0000 7f3e75628740 1 Processor -- start 2024-09-21T05:51:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.009+0000 7f3e75628740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8974560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.009+0000 7f3e75628740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.010+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8974560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.010+0000 7f3e6ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8974560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33320/0 (socket says 172.21.15.28:33320) 2024-09-21T05:51:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.011+0000 7f3e67fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.011+0000 7f3e67fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57418/0 (socket says 172.21.15.28:57418) 2024-09-21T05:51:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.011+0000 7f3e6ce50640 1 -- 172.21.15.28:0/3609395957 learned_addr learned my addr 172.21.15.28:0/3609395957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.011+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f896ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.011+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d5d90 con 0x55f0f85901b0 2024-09-21T05:51:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.012+0000 7f3e6d651640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f896ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.012+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d9280 con 0x55f0f85b9070 2024-09-21T05:51:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.012+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681d6400 con 0x55f0f8598390 2024-09-21T05:51:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.012+0000 7f3e75628740 10 client.0 _ll_register_callbacks cb 0x7f3e581066a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e67fff640 1 -- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 msgr2=0x55f0f896ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e67fff640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f896ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e67fff640 1 -- 172.21.15.28:0/3609395957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 msgr2=0x55f0f8974560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e67fff640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8974560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e67fff640 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f8970740 con 0x55f0f8598390 2024-09-21T05:51:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.013+0000 7f3e6d651640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f896ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.014+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e580cebf0 con 0x55f0f8598390 2024-09-21T05:51:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.014+0000 7f3e67fff640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f0f897e3c0 tx=0x7f3e6000a9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3e5c9fdd7019ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.014+0000 7f3e47fff640 10 client.26332 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:51:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.018+0000 7f3e47fff640 1 -- 172.21.15.28:0/3609395957 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e60005050 con 0x55f0f8598390 2024-09-21T05:51:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.018+0000 7f3e47fff640 1 -- 172.21.15.28:0/3609395957 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3e600051f0 con 0x55f0f8598390 2024-09-21T05:51:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.018+0000 7f3e47fff640 1 -- 172.21.15.28:0/3609395957 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3e60005430 con 0x55f0f8598390 2024-09-21T05:51:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.019+0000 7f3e75628740 10 client.26332 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.019+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=40}) -- 0x55f0f89e84e0 con 0x55f0f8598390 2024-09-21T05:51:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e47fff640 1 -- 172.21.15.28:0/3609395957 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f3e600055d0 con 0x55f0f8598390 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e75628740 10 client.26332 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e75628740 10 client.26332 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e75628740 20 client.26332 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f854cda0 0x55f0f89eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.022+0000 7f3e75628740 4 client.26332 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.023+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3e580cebf0 con 0x55f0f854cda0 2024-09-21T05:51:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.023+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f854cda0 0x55f0f89eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.024+0000 7f3e6ce50640 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f854cda0 0x55f0f89eaad0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4bfd7500aa246830 server_cookie=77f58024cd8bbcfd in_seq=0 out_seq=0 2024-09-21T05:51:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.024+0000 7f3e47fff640 10 client.26332 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:51:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.024+0000 7f3e47fff640 1 -- 172.21.15.28:0/3609395957 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3e580cebf0 con 0x55f0f854cda0 2024-09-21T05:51:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.024+0000 7f3e47fff640 10 client.26332 handle_command_reply: tid=0 2024-09-21T05:51:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.024+0000 7f3e75628740 1 client.26332 shutdown 2024-09-21T05:51:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.025+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 msgr2=0x55f0f896fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.025+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f0f897e3c0 tx=0x7f3e6000a9c0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 shutdown_connections 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55f0f854cda0 0x55f0f89eaad0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0f85b9070 0x55f0f896ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e47fff640 0 client.26332 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f0f8598390 0x55f0f896fa10 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f0f897e3c0 tx=0x7f3e6000a9c0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 --2- 172.21.15.28:0/3609395957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0f85901b0 0x55f0f8974560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.026+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 >> 172.21.15.28:0/3609395957 conn(0x55f0f85ad460 msgr2=0x55f0f896d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.028+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 shutdown_connections 2024-09-21T05:51:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.028+0000 7f3e75628740 1 -- 172.21.15.28:0/3609395957 wait complete. 2024-09-21T05:51:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.028+0000 7f3e75628740 20 client.26332 trim_cache size 0 max 16384 2024-09-21T05:51:44.032 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:51:44.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 msgr2=0x7f3e500b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:44.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 0x7f3e500b8d10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e680865d0 tx=0x7f3e5c00f040 comp rx=0 tx=0).stop 2024-09-21T05:51:44.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 msgr2=0x7f3e68080640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3e58005ab0 tx=0x7f3e5800a690 comp rx=0 tx=0).stop 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 shutdown_connections 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e68110490 0x7f3e680810c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e6810fb40 0x7f3e68080b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e500b6800 0x7f3e500b8d10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 --2- 172.21.15.28:0/3043518385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e6810baf0 0x7f3e68080640 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.031+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 >> 172.21.15.28:0/3043518385 conn(0x7f3e68070ec0 msgr2=0x7f3e6810d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.032+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 shutdown_connections 2024-09-21T05:51:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:44.032+0000 7f3e6f0b4640 1 -- 172.21.15.28:0/3043518385 wait complete. 2024-09-21T05:51:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:44 smithi028 ceph-mon[31227]: pgmap v466: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 5.58k op/s 2024-09-21T05:51:44.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:44 smithi181 ceph-mon[34819]: pgmap v466: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 5.58k op/s 2024-09-21T05:51:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:44 smithi129 ceph-mon[34885]: pgmap v466: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 5.58k op/s 2024-09-21T05:51:44.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:51:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.038+0000 7f7e56a67640 1 -- 172.21.15.28:0/2722553988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5010a9e0 msgr2=0x7f7e5010adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.038+0000 7f7e56a67640 1 --2- 172.21.15.28:0/2722553988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5010a9e0 0x7f7e5010adf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009900 tx=0x7f7e4002a3f0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.040+0000 7f7e56a67640 1 -- 172.21.15.28:0/2722553988 shutdown_connections 2024-09-21T05:51:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.040+0000 7f7e56a67640 1 --2- 172.21.15.28:0/2722553988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e5011cc30 0x7f7e5011f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.040+0000 7f7e56a67640 1 --2- 172.21.15.28:0/2722553988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5010a9e0 0x7f7e5010adf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.040+0000 7f7e56a67640 1 --2- 172.21.15.28:0/2722553988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e50111f40 0x7f7e5010a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.040+0000 7f7e56a67640 1 -- 172.21.15.28:0/2722553988 >> 172.21.15.28:0/2722553988 conn(0x7f7e50070ec0 msgr2=0x7f7e500712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.041+0000 7f7e56a67640 1 -- 172.21.15.28:0/2722553988 shutdown_connections 2024-09-21T05:51:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.042+0000 7f7e56a67640 1 -- 172.21.15.28:0/2722553988 wait complete. 2024-09-21T05:51:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.043+0000 7f7e56a67640 1 Processor -- start 2024-09-21T05:51:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 -- start start 2024-09-21T05:51:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e50111f40 0x7f7e501137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5011cc30 0x7f7e50113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50122900 con 0x7f7e50111f40 2024-09-21T05:51:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50122780 con 0x7f7e5011cc30 2024-09-21T05:51:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e56a67640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50122a80 con 0x7f7e5010a9e0 2024-09-21T05:51:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.044+0000 7f7e54803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41418/0 (socket says 172.21.15.28:41418) 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 -- 172.21.15.28:0/1541116120 learned_addr learned my addr 172.21.15.28:0/1541116120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e50111f40 0x7f7e501137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e55004640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5011cc30 0x7f7e50113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 -- 172.21.15.28:0/1541116120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5011cc30 msgr2=0x7f7e50113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5011cc30 0x7f7e50113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 -- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e50111f40 msgr2=0x7f7e501137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e50111f40 0x7f7e501137e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.045+0000 7f7e54803640 1 -- 172.21.15.28:0/1541116120 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e50114600 con 0x7f7e5010a9e0 2024-09-21T05:51:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.046+0000 7f7e54803640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c890 tx=0x7f7e4400cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.046+0000 7f7e4dffb640 1 -- 172.21.15.28:0/1541116120 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44004220 con 0x7f7e5010a9e0 2024-09-21T05:51:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.047+0000 7f7e4dffb640 1 -- 172.21.15.28:0/1541116120 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e440043c0 con 0x7f7e5010a9e0 2024-09-21T05:51:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.048+0000 7f7e4dffb640 1 -- 172.21.15.28:0/1541116120 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44004a00 con 0x7f7e5010a9e0 2024-09-21T05:51:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.048+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e501192a0 con 0x7f7e5010a9e0 2024-09-21T05:51:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.048+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e50119760 con 0x7f7e5010a9e0 2024-09-21T05:51:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.057+0000 7f7e4dffb640 1 -- 172.21.15.28:0/1541116120 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7e440079b0 con 0x7f7e5010a9e0 2024-09-21T05:51:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.069+0000 7f7e4dffb640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 0x7f7e380b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.069+0000 7f7e4dffb640 1 -- 172.21.15.28:0/1541116120 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7e440e7640 con 0x7f7e5010a9e0 2024-09-21T05:51:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.070+0000 7f7e5cfdb740 1 Processor -- start 2024-09-21T05:51:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 -- start start 2024-09-21T05:51:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 0x56242d9cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56242d941ea0 con 0x56242d5fb1b0 2024-09-21T05:51:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56242d9312d0 con 0x56242d624070 2024-09-21T05:51:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.074+0000 7f7e5cfdb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56242d9350e0 con 0x56242d603390 2024-09-21T05:51:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.076+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 0x7f7e380b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.076+0000 7f7e55004640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 0x56242d9cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.076+0000 7f7e55004640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 0x56242d9cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41428/0 (socket says 172.21.15.28:41428) 2024-09-21T05:51:46.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.076+0000 7f7e55004640 1 -- 172.21.15.28:0/4272370163 learned_addr learned my addr 172.21.15.28:0/4272370163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:46.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.077+0000 7f7e54803640 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.080+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.086+0000 7f7e54803640 1 -- 172.21.15.28:0/4272370163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 msgr2=0x56242d9cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.086+0000 7f7e54803640 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 0x56242d9cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.086+0000 7f7e54803640 1 -- 172.21.15.28:0/4272370163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 msgr2=0x56242d9cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.086+0000 7f7e54803640 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.086+0000 7f7e54803640 1 -- 172.21.15.28:0/4272370163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242d68f650 con 0x56242d624070 2024-09-21T05:51:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.089+0000 7f7e54803640 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9cb6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e440e4db0 tx=0x7f7e440a9480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc03c418c69a07d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.089+0000 7f7e36ffd640 1 -- 172.21.15.28:0/4272370163 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440b1f90 con 0x56242d624070 2024-09-21T05:51:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.090+0000 7f7e36ffd640 1 -- 172.21.15.28:0/4272370163 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e440b6ed0 con 0x56242d624070 2024-09-21T05:51:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.090+0000 7f7e36ffd640 1 -- 172.21.15.28:0/4272370163 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440ac180 con 0x56242d624070 2024-09-21T05:51:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.091+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/4272370163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 msgr2=0x56242d9cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.091+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9cb6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e440e4db0 tx=0x7f7e440a9480 comp rx=0 tx=0).stop 2024-09-21T05:51:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.091+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 0x7f7e380b8b30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7e4002a9e0 tx=0x7f7e40000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.093+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/4272370163 shutdown_connections 2024-09-21T05:51:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.093+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d603390 0x56242d9cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.093+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.093+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/4272370163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9cb6a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.093+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/4272370163 >> 172.21.15.28:0/4272370163 conn(0x56242d618460 msgr2=0x56242d9c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.094+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/4272370163 shutdown_connections 2024-09-21T05:51:46.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.095+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/4272370163 wait complete. 2024-09-21T05:51:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.095+0000 7f7e5cfdb740 1 -- start start 2024-09-21T05:51:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.096+0000 7f7e5cfdb740 1 Processor -- start 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 0x56242d9d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9d7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e54803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 0x56242d9d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e54803640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 0x56242d9d6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41430/0 (socket says 172.21.15.28:41430) 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e54803640 1 -- 172.21.15.28:0/1084949653 learned_addr learned my addr 172.21.15.28:0/1084949653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9e3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.097+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56242d792700 con 0x56242d603390 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56242d8cc160 con 0x56242d624070 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56242d9d9e20 con 0x56242d5fb1b0 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9e3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e5cfdb740 10 client.0 _ll_register_callbacks cb 0x56242d8e59e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 -- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 msgr2=0x56242d9d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 0x56242d9d6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 -- 172.21.15.28:0/1084949653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 msgr2=0x56242d9d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9d7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.098+0000 7f7e55004640 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242d8f9c90 con 0x56242d624070 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.099+0000 7f7e55004640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9e3110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e48010a30 tx=0x7f7e48010ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8353e86be63283d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.099+0000 7f7e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.100+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800fdd0 con 0x56242d624070 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.100+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e440b6ed0 con 0x56242d624070 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.100+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7e48003970 con 0x56242d624070 2024-09-21T05:51:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.100+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7e48003b70 con 0x56242d624070 2024-09-21T05:51:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.106+0000 7f7e5cfdb740 10 client.26318 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.106+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56242d9e5bd0 con 0x56242d624070 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.106+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e440b6ed0 con 0x56242d624070 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.107+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f7e48021020 con 0x56242d624070 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 10 client.26318 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 20 client.26318 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5e29f0 0x56242d9e9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 4 client.26318 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56242d943df0 con 0x56242d5e29f0 2024-09-21T05:51:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.109+0000 7f7e54803640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5e29f0 0x56242d9e9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.110+0000 7f7e54803640 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5e29f0 0x56242d9e9070 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d5b2ac7834a0c1 server_cookie=848d329391b46ca5 in_seq=0 out_seq=0 2024-09-21T05:51:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.110+0000 7f7e2ffff640 10 client.26318 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.111+0000 7f7e2ffff640 1 -- 172.21.15.28:0/1084949653 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56242d943df0 con 0x56242d5e29f0 2024-09-21T05:51:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.111+0000 7f7e2ffff640 10 client.26318 handle_command_reply: tid=0 2024-09-21T05:51:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 client.26318 shutdown 2024-09-21T05:51:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 msgr2=0x56242d9e3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9e3110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e48010a30 tx=0x7f7e48010ef0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 shutdown_connections 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5e29f0 0x56242d9e9070 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d624070 0x56242d9e3110 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9d7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e2ffff640 0 client.26318 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/1084949653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d5fb1b0 0x56242d9d6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.114+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 >> 172.21.15.28:0/1084949653 conn(0x56242d618460 msgr2=0x56242d9ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.116+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 shutdown_connections 2024-09-21T05:51:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.116+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/1084949653 wait complete. 2024-09-21T05:51:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.116+0000 7f7e5cfdb740 20 client.26318 trim_cache size 0 max 16384 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.142+0000 7f7e5cfdb740 1 Processor -- start 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.142+0000 7f7e5cfdb740 1 -- start start 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 0x56242d9d8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e54803640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 0x56242d9d8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43046/0 (socket says 172.21.15.28:43046) 2024-09-21T05:51:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e4ffff640 1 -- 172.21.15.28:0/220929811 learned_addr learned my addr 172.21.15.28:0/220929811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9e3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c6c00 con 0x56242d603390 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c87a0 con 0x56242d5fb1b0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c72d0 con 0x56242d624070 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.143+0000 7f7e55004640 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 -- 172.21.15.28:0/220929811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 msgr2=0x56242d9e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9e3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 -- 172.21.15.28:0/220929811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 msgr2=0x56242d9ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 -- 172.21.15.28:0/220929811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242d9cddf0 con 0x56242d5fb1b0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.144+0000 7f7e54803640 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 0x56242d9d8ca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7e44007590 tx=0x7f7e440a9bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ddb317c4be78cd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.145+0000 7f7e367fc640 1 -- 172.21.15.28:0/220929811 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440b1f90 con 0x56242d5fb1b0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.145+0000 7f7e367fc640 1 -- 172.21.15.28:0/220929811 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e44007ca0 con 0x56242d5fb1b0 2024-09-21T05:51:46.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.145+0000 7f7e367fc640 1 -- 172.21.15.28:0/220929811 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440ac3e0 con 0x56242d5fb1b0 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.146+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 msgr2=0x56242d9d8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.146+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 0x56242d9d8ca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7e44007590 tx=0x7f7e440a9bb0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 shutdown_connections 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d603390 0x56242d9ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/220929811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d5fb1b0 0x56242d9d8ca0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 >> 172.21.15.28:0/220929811 conn(0x56242d618460 msgr2=0x56242d9cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 shutdown_connections 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.147+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/220929811 wait complete. 2024-09-21T05:51:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.148+0000 7f7e5cfdb740 1 -- start start 2024-09-21T05:51:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.148+0000 7f7e5cfdb740 1 Processor -- start 2024-09-21T05:51:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.149+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.149+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 0x56242d9dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e5cfdb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e54803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e54803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43060/0 (socket says 172.21.15.28:43060) 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e54803640 1 -- 172.21.15.28:0/2673866648 learned_addr learned my addr 172.21.15.28:0/2673866648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 0x56242d9dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c80d0 con 0x56242d5fb1b0 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e55004640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c79d0 con 0x56242d603390 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501c4be0 con 0x56242d624070 2024-09-21T05:51:46.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.150+0000 7f7e5cfdb740 10 client.0 _ll_register_callbacks cb 0x7f7e48007a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 -- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 msgr2=0x56242d9ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9ddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 -- 172.21.15.28:0/2673866648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 msgr2=0x56242d9dd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 0x56242d9dd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242da52a40 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e54803640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e440a9960 tx=0x7f7e440a9990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85645f6e6bdb75d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.151+0000 7f7e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440a9100 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 0x56242d9dd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7e440ac710 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e55004640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e44007ca0 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7e44007e50 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e5cfdb740 10 client.16746 fetch_fsmap learned FSMap version 40 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.152+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56242da53cc0 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e44007ca0 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f7e440ac180 con 0x56242d5fb1b0 2024-09-21T05:51:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 10 client.16746 fetch_fsmap finished waiting for FSMap version 40 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 20 client.16746 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5b7da0 0x56242da55d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 4 client.16746 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56242d93f8a0 con 0x56242d5b7da0 2024-09-21T05:51:46.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.153+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5b7da0 0x56242da55d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e2ffff640 10 client.16746 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e4ffff640 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5b7da0 0x56242da55d50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a45c89b4e083c6 server_cookie=67c670ca8b15ecba in_seq=0 out_seq=0 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e2ffff640 1 -- 172.21.15.28:0/2673866648 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56242d93f8a0 con 0x56242d5b7da0 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e2ffff640 10 client.16746 handle_command_reply: tid=0 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e5cfdb740 1 client.16746 shutdown 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.154+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 msgr2=0x56242d9dd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e440a9960 tx=0x7f7e440a9990 comp rx=0 tx=0).stop 2024-09-21T05:51:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 shutdown_connections 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56242d5b7da0 0x56242da55d50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56242d624070 0x56242d9ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e2ffff640 0 client.16746 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56242d603390 0x56242d9dd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 --2- 172.21.15.28:0/2673866648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242d5fb1b0 0x56242d9dd070 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.155+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 >> 172.21.15.28:0/2673866648 conn(0x56242d618460 msgr2=0x56242d9cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.156+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 shutdown_connections 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.156+0000 7f7e5cfdb740 1 -- 172.21.15.28:0/2673866648 wait complete. 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.156+0000 7f7e5cfdb740 20 client.16746 trim_cache size 0 max 16384 2024-09-21T05:51:46.160 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.157+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 msgr2=0x7f7e380b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.157+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 0x7f7e380b8b30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7e4002a9e0 tx=0x7f7e40000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.157+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 msgr2=0x7f7e5011a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.157+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c890 tx=0x7f7e4400cd50 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 shutdown_connections 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e5011cc30 0x7f7e50113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7e380b6620 0x7f7e380b8b30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e50111f40 0x7f7e501137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 --2- 172.21.15.28:0/1541116120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e5010a9e0 0x7f7e5011a510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 >> 172.21.15.28:0/1541116120 conn(0x7f7e50070ec0 msgr2=0x7f7e50071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 shutdown_connections 2024-09-21T05:51:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:46.158+0000 7f7e56a67640 1 -- 172.21.15.28:0/1541116120 wait complete. 2024-09-21T05:51:46.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:46 smithi181 ceph-mon[34819]: pgmap v467: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 MiB/s wr, 6.49k op/s 2024-09-21T05:51:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:46 smithi028 ceph-mon[31227]: pgmap v467: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 MiB/s wr, 6.49k op/s 2024-09-21T05:51:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:46 smithi129 ceph-mon[34885]: pgmap v467: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 MiB/s wr, 6.49k op/s 2024-09-21T05:51:48.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:48 smithi181 ceph-mon[34819]: pgmap v468: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 MiB/s wr, 6.39k op/s 2024-09-21T05:51:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:48 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:48 smithi028 ceph-mon[31227]: pgmap v468: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 MiB/s wr, 6.39k op/s 2024-09-21T05:51:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:48 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:48 smithi129 ceph-mon[34885]: pgmap v468: 97 pgs: 97 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 MiB/s wr, 6.39k op/s 2024-09-21T05:51:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:48 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:51:48.776 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:51:48.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:51:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:50.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:50 smithi028 ceph-mon[31227]: pgmap v469: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 8.18k op/s 2024-09-21T05:51:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.392+0000 7f0ece193640 1 -- 172.21.15.28:0/2860114704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 msgr2=0x7f0ec810a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.392+0000 7f0ece193640 1 --2- 172.21.15.28:0/2860114704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec810a3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc02a420 comp rx=0 tx=0).stop 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 -- 172.21.15.28:0/2860114704 shutdown_connections 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 --2- 172.21.15.28:0/2860114704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec811f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 --2- 172.21.15.28:0/2860114704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec810a9e0 0x7f0ec810adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 --2- 172.21.15.28:0/2860114704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec810a3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 -- 172.21.15.28:0/2860114704 >> 172.21.15.28:0/2860114704 conn(0x7f0ec8070ec0 msgr2=0x7f0ec80712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 -- 172.21.15.28:0/2860114704 shutdown_connections 2024-09-21T05:51:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.397+0000 7f0ece193640 1 -- 172.21.15.28:0/2860114704 wait complete. 2024-09-21T05:51:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.398+0000 7f0ece193640 1 Processor -- start 2024-09-21T05:51:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 -- start start 2024-09-21T05:51:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec810a9e0 0x7f0ec81a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec81a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8122960 con 0x7f0ec810a9e0 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81227e0 con 0x7f0ec811cc30 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.399+0000 7f0ece193640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8122ae0 con 0x7f0ec8111f40 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54634/0 (socket says 172.21.15.28:54634) 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 -- 172.21.15.28:0/4122012065 learned_addr learned my addr 172.21.15.28:0/4122012065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec81a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 -- 172.21.15.28:0/4122012065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 msgr2=0x7f0ec81a98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec81a98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 -- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec810a9e0 msgr2=0x7f0ec81a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec810a9e0 0x7f0ec81a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.400+0000 7f0ec7fff640 1 -- 172.21.15.28:0/4122012065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec81af290 con 0x7f0ec811cc30 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.401+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec81a98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:51:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.401+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000cb00 tx=0x7f0ec000a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.401+0000 7f0ec4ff9640 1 -- 172.21.15.28:0/4122012065 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0004b20 con 0x7f0ec811cc30 2024-09-21T05:51:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.401+0000 7f0ec4ff9640 1 -- 172.21.15.28:0/4122012065 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec0004cc0 con 0x7f0ec811cc30 2024-09-21T05:51:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.401+0000 7f0ec4ff9640 1 -- 172.21.15.28:0/4122012065 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000f740 con 0x7f0ec811cc30 2024-09-21T05:51:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.403+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec81af580 con 0x7f0ec811cc30 2024-09-21T05:51:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.403+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec8112380 con 0x7f0ec811cc30 2024-09-21T05:51:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.413+0000 7f0ed4707740 1 Processor -- start 2024-09-21T05:51:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.420+0000 7f0ec4ff9640 1 -- 172.21.15.28:0/4122012065 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f0ec000f8e0 con 0x7f0ec811cc30 2024-09-21T05:51:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.421+0000 7f0ec4ff9640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 0x7f0eb00b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.421+0000 7f0ec4ff9640 1 -- 172.21.15.28:0/4122012065 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0ec00e7a60 con 0x7f0ec811cc30 2024-09-21T05:51:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.422+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 0x7f0eb00b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.422+0000 7f0ed4707740 1 -- start start 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 0x7f0eb00b8d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8071460 tx=0x7f0eb8008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56247fd6ce60 con 0x56247fa4f350 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56247fd5c290 con 0x56247fa2e670 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ed4707740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56247fd600a0 con 0x56247fa26490 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43086/0 (socket says 172.21.15.28:43086) 2024-09-21T05:51:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ec77fe640 1 -- 172.21.15.28:0/832293490 learned_addr learned my addr 172.21.15.28:0/832293490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.423+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 -- 172.21.15.28:0/832293490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 msgr2=0x56247fdf6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 -- 172.21.15.28:0/832293490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 msgr2=0x56247fdf6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 -- 172.21.15.28:0/832293490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247faba930 con 0x56247fa4f350 2024-09-21T05:51:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.424+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc004180 tx=0x7f0ebc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359ad00db7380855 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.429+0000 7f0eadffb640 1 -- 172.21.15.28:0/832293490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc03e030 con 0x56247fa4f350 2024-09-21T05:51:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.429+0000 7f0eadffb640 1 -- 172.21.15.28:0/832293490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ebc004420 con 0x56247fa4f350 2024-09-21T05:51:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.429+0000 7f0eadffb640 1 -- 172.21.15.28:0/832293490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc035040 con 0x56247fa4f350 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 -- 172.21.15.28:0/832293490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 msgr2=0x56247fdf6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc004180 tx=0x7f0ebc000bc0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 -- 172.21.15.28:0/832293490 shutdown_connections 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 --2- 172.21.15.28:0/832293490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdf6430 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.431+0000 7f0ed4707740 1 -- 172.21.15.28:0/832293490 >> 172.21.15.28:0/832293490 conn(0x56247fa43740 msgr2=0x56247fdf4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.435+0000 7f0ed4707740 1 -- 172.21.15.28:0/832293490 shutdown_connections 2024-09-21T05:51:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.437+0000 7f0ed4707740 1 -- 172.21.15.28:0/832293490 wait complete. 2024-09-21T05:51:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.437+0000 7f0ed4707740 1 -- start start 2024-09-21T05:51:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ed4707740 1 Processor -- start 2024-09-21T05:51:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa2e670 0x56247fe026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe07e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.438+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43102/0 (socket says 172.21.15.28:43102) 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ed4707740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56247fbbd890 con 0x56247fa26490 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ed4707740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56247fcf6d40 con 0x56247fa4f350 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ed4707740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56247fe04bb0 con 0x56247fa2e670 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa2e670 0x56247fe026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 -- 172.21.15.28:0/1824092976 learned_addr learned my addr 172.21.15.28:0/1824092976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe07e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ed4707740 10 client.0 _ll_register_callbacks cb 0x56247fd5ac80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 -- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa2e670 msgr2=0x56247fe026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa2e670 0x56247fe026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 -- 172.21.15.28:0/1824092976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 msgr2=0x56247fe07e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe07e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.439+0000 7f0ec77fe640 1 -- 172.21.15.28:0/1824092976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247fe084e0 con 0x56247fa26490 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.447+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc031950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46360e19f4d8b022 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.447+0000 7f0ea6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.447+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc035050 con 0x56247fa26490 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.448+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0ebc031cc0 con 0x56247fa26490 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.448+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ebc004420 con 0x56247fa26490 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.448+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f0ebc03e040 con 0x56247fa26490 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.448+0000 7f0ed4707740 10 client.16752 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.448+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x56247fe10e50 con 0x56247fa26490 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.450+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ebc004420 con 0x56247fa26490 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f0ebc055580 con 0x56247fa26490 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 10 client.16752 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 20 client.16752 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247fa0dcd0 0x56247fe141b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 4 client.16752 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.451+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56247fd6edb0 con 0x56247fa0dcd0 2024-09-21T05:51:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.454+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247fa0dcd0 0x56247fe141b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.462+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247fa0dcd0 0x56247fe141b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e450c140523f58d4 server_cookie=5bf418347cf34198 in_seq=0 out_seq=0 2024-09-21T05:51:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.462+0000 7f0ea6ffd640 10 client.16752 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.463+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1824092976 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56247fd6edb0 con 0x56247fa0dcd0 2024-09-21T05:51:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.463+0000 7f0ea6ffd640 10 client.16752 handle_command_reply: tid=0 2024-09-21T05:51:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.463+0000 7f0ed4707740 1 client.16752 shutdown 2024-09-21T05:51:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.464+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 msgr2=0x56247fe02100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.464+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc031950 comp rx=0 tx=0).stop 2024-09-21T05:51:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 shutdown_connections 2024-09-21T05:51:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247fa0dcd0 0x56247fe141b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe07e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa2e670 0x56247fe026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1824092976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa26490 0x56247fe02100 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 >> 172.21.15.28:0/1824092976 conn(0x56247fa43740 msgr2=0x56247fdf9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.465+0000 7f0ea6ffd640 0 client.16752 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.468+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 shutdown_connections 2024-09-21T05:51:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.468+0000 7f0ed4707740 1 -- 172.21.15.28:0/1824092976 wait complete. 2024-09-21T05:51:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.468+0000 7f0ed4707740 20 client.16752 trim_cache size 0 max 16384 2024-09-21T05:51:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.507+0000 7f0ed4707740 1 Processor -- start 2024-09-21T05:51:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.508+0000 7f0ed4707740 1 -- start start 2024-09-21T05:51:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.508+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.508+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ec77fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41486/0 (socket says 172.21.15.28:41486) 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ec77fe640 1 -- 172.21.15.28:0/1095425352 learned_addr learned my addr 172.21.15.28:0/1095425352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdfb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8075710 con 0x56247fa4f350 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8077280 con 0x56247fa2e670 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8075de0 con 0x56247fa26490 2024-09-21T05:51:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.509+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdfb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1095425352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 msgr2=0x56247fdf9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1095425352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 msgr2=0x56247fdf9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1095425352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247fdfd7b0 con 0x56247fa4f350 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec6ffd640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.510+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:51:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.511+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdfb510 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00e5170 tx=0x7f0ec00ad7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd7be013a9823a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.512+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1095425352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00b21d0 con 0x56247fa4f350 2024-09-21T05:51:50.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.512+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1095425352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec005a030 con 0x56247fa4f350 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.512+0000 7f0ead7fa640 1 -- 172.21.15.28:0/1095425352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00a94a0 con 0x56247fa4f350 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 msgr2=0x56247fdfb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdfb510 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00e5170 tx=0x7f0ec00ad7e0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 shutdown_connections 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa4f350 0x56247fdfb510 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa2e670 0x56247fdf9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1095425352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fdf9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.513+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 >> 172.21.15.28:0/1095425352 conn(0x56247fa43740 msgr2=0x56247fe007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.514+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 shutdown_connections 2024-09-21T05:51:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.515+0000 7f0ed4707740 1 -- 172.21.15.28:0/1095425352 wait complete. 2024-09-21T05:51:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.515+0000 7f0ed4707740 1 -- start start 2024-09-21T05:51:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.516+0000 7f0ed4707740 1 Processor -- start 2024-09-21T05:51:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.517+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fe0c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.517+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa2e670 0x56247fe0cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.517+0000 7f0ed4707740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.517+0000 7f0ed4707740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8076bb0 con 0x56247fa2e670 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ed4707740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec80764b0 con 0x56247fa4f350 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ed4707740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec810adf0 con 0x56247fa26490 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ec7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ec7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54670/0 (socket says 172.21.15.28:54670) 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1132490625 learned_addr learned my addr 172.21.15.28:0/1132490625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.518+0000 7f0ed4707740 10 client.0 _ll_register_callbacks cb 0x7f0ebc0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 msgr2=0x56247fe0c660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fe0c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1132490625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa2e670 msgr2=0x56247fe0cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa2e670 0x56247fe0cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 -- 172.21.15.28:0/1132490625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247fe7df20 con 0x56247fa4f350 2024-09-21T05:51:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.519+0000 7f0ec7fff640 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0096040 tx=0x7f0ec00acd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edfe702dfefa0c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.520+0000 7f0ea6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.522+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ec005a030 con 0x56247fa4f350 2024-09-21T05:51:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.524+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1132490625 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00c7070 con 0x56247fa4f350 2024-09-21T05:51:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.524+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1132490625 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f0ec00ac5c0 con 0x56247fa4f350 2024-09-21T05:51:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.525+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1132490625 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f0ec0098040 con 0x56247fa4f350 2024-09-21T05:51:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.528+0000 7f0ed4707740 10 client.26333 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.528+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=41}) -- 0x56247fe7f050 con 0x56247fa4f350 2024-09-21T05:51:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1132490625 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f0ec00cf030 con 0x56247fa4f350 2024-09-21T05:51:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 10 client.26333 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 20 client.26333 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247f9e3080 0x56247fe81640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 4 client.26333 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:51:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.529+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f0ec005a030 con 0x56247f9e3080 2024-09-21T05:51:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.534+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247f9e3080 0x56247fe81640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:50.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:50 smithi181 ceph-mon[34819]: pgmap v469: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 8.18k op/s 2024-09-21T05:51:50.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.540+0000 7f0ec77fe640 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247f9e3080 0x56247fe81640 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5646737dfd783b84 server_cookie=769da97ef750bda0 in_seq=0 out_seq=0 2024-09-21T05:51:50.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.541+0000 7f0ea6ffd640 10 client.26333 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:50.548 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "cabb93db-81c7-44a8-a5df-72df2a2e8078", 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ea6ffd640 1 -- 172.21.15.28:0/1132490625 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f0ec005a030 con 0x56247f9e3080 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ea6ffd640 10 client.26333 handle_command_reply: tid=0 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 client.26333 shutdown 2024-09-21T05:51:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 msgr2=0x56247fe7d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0096040 tx=0x7f0ec00acd80 comp rx=0 tx=0).stop 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 shutdown_connections 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56247f9e3080 0x56247fe81640 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247fa4f350 0x56247fe7d930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56247fa2e670 0x56247fe0cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 --2- 172.21.15.28:0/1132490625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56247fa26490 0x56247fe0c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ea6ffd640 0 client.26333 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.545+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 >> 172.21.15.28:0/1132490625 conn(0x56247fa43740 msgr2=0x56247fdf9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.546+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 shutdown_connections 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.546+0000 7f0ed4707740 1 -- 172.21.15.28:0/1132490625 wait complete. 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.546+0000 7f0ed4707740 20 client.26333 trim_cache size 0 max 16384 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 msgr2=0x7f0eb00b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 0x7f0eb00b8d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8071460 tx=0x7f0eb8008040 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 msgr2=0x7f0ec81aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000cb00 tx=0x7f0ec000a250 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 shutdown_connections 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec811cc30 0x7f0ec81aeb90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec8111f40 0x7f0ec81a98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f0eb00b6850 0x7f0eb00b8d60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 --2- 172.21.15.28:0/4122012065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec810a9e0 0x7f0ec81a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 >> 172.21.15.28:0/4122012065 conn(0x7f0ec8070ec0 msgr2=0x7f0ec811e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 shutdown_connections 2024-09-21T05:51:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:50.547+0000 7f0ece193640 1 -- 172.21.15.28:0/4122012065 wait complete. 2024-09-21T05:51:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:50 smithi129 ceph-mon[34885]: pgmap v469: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 8.18k op/s 2024-09-21T05:51:51.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:51:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:51 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:51:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:51 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:51 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:51:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:51 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:51:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:51 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:51:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:51 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:51:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:51 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:51 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:51:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:51 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:51:51.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:51 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:51:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:51 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:51:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:51 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:51 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:51:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:51 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:51:51.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:51 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:51:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:51:52.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:51:52.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:51:52.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:51:52.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: pgmap v470: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 30 KiB/s rd, 37 MiB/s wr, 8.21k op/s 2024-09-21T05:51:52.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:51:52.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:51:52.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: pgmap v470: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 30 KiB/s rd, 37 MiB/s wr, 8.21k op/s 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:51:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: pgmap v470: 97 pgs: 97 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 30 KiB/s rd, 37 MiB/s wr, 8.21k op/s 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:51:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 -- 172.21.15.28:0/1162655146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10aee0 msgr2=0x7f8f0c10b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 --2- 172.21.15.28:0/1162655146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c10b2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b9f0 tx=0x7f8f0802c0f0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 -- 172.21.15.28:0/1162655146 shutdown_connections 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 --2- 172.21.15.28:0/1162655146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 0x7f8f0c1174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 --2- 172.21.15.28:0/1162655146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10e180 0x7f8f0c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 --2- 172.21.15.28:0/1162655146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c10b2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.686+0000 7f8f134f7640 1 -- 172.21.15.28:0/1162655146 >> 172.21.15.28:0/1162655146 conn(0x7f8f0c070ec0 msgr2=0x7f8f0c0712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.687+0000 7f8f134f7640 1 -- 172.21.15.28:0/1162655146 shutdown_connections 2024-09-21T05:51:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.688+0000 7f8f134f7640 1 -- 172.21.15.28:0/1162655146 wait complete. 2024-09-21T05:51:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.689+0000 7f8f134f7640 1 Processor -- start 2024-09-21T05:51:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 -- start start 2024-09-21T05:51:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c1164c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 0x7f8f0c1ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c119710 con 0x7f8f0c10ead0 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c119590 con 0x7f8f0c10e180 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.691+0000 7f8f134f7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c119890 con 0x7f8f0c10aee0 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.694+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.694+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54694/0 (socket says 172.21.15.28:54694) 2024-09-21T05:51:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.694+0000 7f8f10a92640 1 -- 172.21.15.28:0/3293436940 learned_addr learned my addr 172.21.15.28:0/3293436940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.695+0000 7f8f11a94640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 0x7f8f0c1ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.695+0000 7f8f11293640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c1164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.696+0000 7f8f10a92640 1 -- 172.21.15.28:0/3293436940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10aee0 msgr2=0x7f8f0c1164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.696+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c1164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.696+0000 7f8f10a92640 1 -- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 msgr2=0x7f8f0c1ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.696+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 0x7f8f0c1ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.696+0000 7f8f10a92640 1 -- 172.21.15.28:0/3293436940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0c1ad030 con 0x7f8f0c10e180 2024-09-21T05:51:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.697+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d7a0 tx=0x7f8efc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.699+0000 7f8f027fc640 1 -- 172.21.15.28:0/3293436940 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc017070 con 0x7f8f0c10e180 2024-09-21T05:51:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.699+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c1ad320 con 0x7f8f0c10e180 2024-09-21T05:51:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.699+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c11d310 con 0x7f8f0c10e180 2024-09-21T05:51:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.699+0000 7f8f027fc640 1 -- 172.21.15.28:0/3293436940 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8efc009c30 con 0x7f8f0c10e180 2024-09-21T05:51:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.701+0000 7f8f027fc640 1 -- 172.21.15.28:0/3293436940 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc00f620 con 0x7f8f0c10e180 2024-09-21T05:51:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.703+0000 7f8f19a6b740 1 Processor -- start 2024-09-21T05:51:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.703+0000 7f8f19a6b740 1 -- start start 2024-09-21T05:51:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c677b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c67cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca36bdde50 con 0x55ca3689f390 2024-09-21T05:51:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca36bcd280 con 0x55ca368c0070 2024-09-21T05:51:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ca36bd1090 con 0x55ca368971b0 2024-09-21T05:51:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.706+0000 7f8f11a94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.707+0000 7f8f11a94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54704/0 (socket says 172.21.15.28:54704) 2024-09-21T05:51:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.707+0000 7f8f11a94640 1 -- 172.21.15.28:0/392915001 learned_addr learned my addr 172.21.15.28:0/392915001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11293640 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c677b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f027fc640 1 -- 172.21.15.28:0/3293436940 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8efc01a070 con 0x7f8f0c10e180 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 -- 172.21.15.28:0/392915001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 msgr2=0x55ca36c677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c677b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 -- 172.21.15.28:0/392915001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 msgr2=0x55ca36c67cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c67cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 -- 172.21.15.28:0/392915001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca3692b650 con 0x55ca368c0070 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.708+0000 7f8f11a94640 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400af60 tx=0x7f8f0400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0dc941f7f91b236 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.709+0000 7f8f027fc640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 0x7f8ef80b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.710+0000 7f8f027fc640 1 -- 172.21.15.28:0/3293436940 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8efc0e6fd0 con 0x7f8f0c10e180 2024-09-21T05:51:52.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.709+0000 7f8ef77fe640 1 -- 172.21.15.28:0/392915001 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04016610 con 0x55ca368c0070 2024-09-21T05:51:52.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.710+0000 7f8ef77fe640 1 -- 172.21.15.28:0/392915001 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f0401b070 con 0x55ca368c0070 2024-09-21T05:51:52.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.710+0000 7f8ef77fe640 1 -- 172.21.15.28:0/392915001 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400ba60 con 0x55ca368c0070 2024-09-21T05:51:52.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.713+0000 7f8f19a6b740 1 -- 172.21.15.28:0/392915001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 msgr2=0x55ca36c67270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.713+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400af60 tx=0x7f8f0400a620 comp rx=0 tx=0).stop 2024-09-21T05:51:52.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.715+0000 7f8f19a6b740 1 -- 172.21.15.28:0/392915001 shutdown_connections 2024-09-21T05:51:52.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.715+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c67cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.715+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c677b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.715+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/392915001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c67270 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.715+0000 7f8f19a6b740 1 -- 172.21.15.28:0/392915001 >> 172.21.15.28:0/392915001 conn(0x55ca368b4460 msgr2=0x55ca36c65160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.716+0000 7f8f11293640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 0x7f8ef80b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.717+0000 7f8f11293640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 0x7f8ef80b8d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b9c0 tx=0x7f8f0802e040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.717+0000 7f8f19a6b740 1 -- 172.21.15.28:0/392915001 shutdown_connections 2024-09-21T05:51:52.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.717+0000 7f8f19a6b740 1 -- 172.21.15.28:0/392915001 wait complete. 2024-09-21T05:51:52.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.718+0000 7f8f19a6b740 1 -- start start 2024-09-21T05:51:52.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.718+0000 7f8f19a6b740 1 Processor -- start 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c72980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c72ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca36a2e700 con 0x55ca3689f390 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca36b68110 con 0x55ca368c0070 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ca36c75a50 con 0x55ca368971b0 2024-09-21T05:51:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.720+0000 7f8f19a6b740 10 client.0 _ll_register_callbacks cb 0x55ca36b81990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54708/0 (socket says 172.21.15.28:54708) 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 -- 172.21.15.28:0/3466485845 learned_addr learned my addr 172.21.15.28:0/3466485845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c72ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 -- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 msgr2=0x55ca36c72980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c72980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 -- 172.21.15.28:0/3466485845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 msgr2=0x55ca36c72ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c72ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.724+0000 7f8f11a94640 1 -- 172.21.15.28:0/3466485845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca36b95c40 con 0x55ca368c0070 2024-09-21T05:51:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.726+0000 7f8f11a94640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000f80 tx=0x7f8f0400caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8408afc29c75a8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.726+0000 7f8ef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:52.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.727+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400cea0 con 0x55ca368c0070 2024-09-21T05:51:52.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.727+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f0401b070 con 0x55ca368c0070 2024-09-21T05:51:52.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.727+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8f04005240 con 0x55ca368c0070 2024-09-21T05:51:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.727+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8f0400abd0 con 0x55ca368c0070 2024-09-21T05:51:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.727+0000 7f8f19a6b740 10 client.26345 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.728+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ca36c817c0 con 0x55ca368c0070 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.728+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f0401b070 con 0x55ca368c0070 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f8f04016740 con 0x55ca368c0070 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 10 client.26345 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 20 client.26345 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca3687e9f0 0x55ca36c85050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 4 client.26345 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.730+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca36bdfda0 con 0x55ca3687e9f0 2024-09-21T05:51:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.742+0000 7f8f11293640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca3687e9f0 0x55ca36c85050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.743+0000 7f8ef4ff9640 10 client.26345 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.743+0000 7f8f11293640 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca3687e9f0 0x55ca36c85050 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b0471db159f11 server_cookie=e728e4b6bedf3f80 in_seq=0 out_seq=0 2024-09-21T05:51:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.748+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/3466485845 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ca36bdfda0 con 0x55ca3687e9f0 2024-09-21T05:51:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.748+0000 7f8ef4ff9640 10 client.26345 handle_command_reply: tid=0 2024-09-21T05:51:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.748+0000 7f8f19a6b740 1 client.26345 shutdown 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 msgr2=0x55ca36c7eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000f80 tx=0x7f8f0400caa0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 shutdown_connections 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca3687e9f0 0x55ca36c85050 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36c7eda0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c72ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3466485845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c72980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.749+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 >> 172.21.15.28:0/3466485845 conn(0x55ca368b4460 msgr2=0x55ca36c6a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.750+0000 7f8ef4ff9640 0 client.26345 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.750+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 shutdown_connections 2024-09-21T05:51:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.750+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3466485845 wait complete. 2024-09-21T05:51:52.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.750+0000 7f8f19a6b740 20 client.26345 trim_cache size 0 max 16384 2024-09-21T05:51:52.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.787+0000 7f8f19a6b740 1 Processor -- start 2024-09-21T05:51:52.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.787+0000 7f8f19a6b740 1 -- start start 2024-09-21T05:51:52.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.788+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 0x55ca36c65320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.788+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.788+0000 7f8f11293640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 0x55ca36c65320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.788+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca368c0070 0x55ca36c63e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f11293640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 0x55ca36c65320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54724/0 (socket says 172.21.15.28:54724) 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:41536/0 (socket says 172.21.15.28:41536) 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f11293640 1 -- 172.21.15.28:0/3235205071 learned_addr learned my addr 172.21.15.28:0/3235205071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef000e790 con 0x55ca368c0070 2024-09-21T05:51:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef0010300 con 0x55ca368971b0 2024-09-21T05:51:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef000ee60 con 0x55ca3689f390 2024-09-21T05:51:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.789+0000 7f8f11a94640 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca368c0070 0x55ca36c63e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.790+0000 7f8f10a92640 1 -- 172.21.15.28:0/3235205071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 msgr2=0x55ca36c65320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.790+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 0x55ca36c65320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.790+0000 7f8f10a92640 1 -- 172.21.15.28:0/3235205071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca368c0070 msgr2=0x55ca36c63e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.790+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca368c0070 0x55ca36c63e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.790+0000 7f8f10a92640 1 -- 172.21.15.28:0/3235205071 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca36c5f360 con 0x55ca3689f390 2024-09-21T05:51:52.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.796+0000 7f8f10a92640 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0e46e0 tx=0x7f8efc0acc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6582ff75e098a8d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.798+0000 7f8ef6ffd640 1 -- 172.21.15.28:0/3235205071 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0b0070 con 0x55ca3689f390 2024-09-21T05:51:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.798+0000 7f8ef6ffd640 1 -- 172.21.15.28:0/3235205071 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8efc0ac8f0 con 0x55ca3689f390 2024-09-21T05:51:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.798+0000 7f8ef6ffd640 1 -- 172.21.15.28:0/3235205071 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc021040 con 0x55ca3689f390 2024-09-21T05:51:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.799+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 msgr2=0x55ca36c68c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.800+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0e46e0 tx=0x7f8efc0acc70 comp rx=0 tx=0).stop 2024-09-21T05:51:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.801+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 shutdown_connections 2024-09-21T05:51:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.801+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca368c0070 0x55ca36c63e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.801+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca3689f390 0x55ca36c68c60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.801+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/3235205071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368971b0 0x55ca36c65320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.801+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 >> 172.21.15.28:0/3235205071 conn(0x55ca368b4460 msgr2=0x55ca36c63700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.804+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 shutdown_connections 2024-09-21T05:51:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.804+0000 7f8f19a6b740 1 -- 172.21.15.28:0/3235205071 wait complete. 2024-09-21T05:51:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.805+0000 7f8f19a6b740 1 -- start start 2024-09-21T05:51:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.805+0000 7f8f19a6b740 1 Processor -- start 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.806+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c76720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.806+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.806+0000 7f8f19a6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36ce7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.807+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef000fc30 con 0x55ca3689f390 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.807+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef000f530 con 0x55ca368c0070 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.807+0000 7f8f19a6b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef000c770 con 0x55ca368971b0 2024-09-21T05:51:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.807+0000 7f8f19a6b740 10 client.0 _ll_register_callbacks cb 0x7f8f0400ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.810+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.810+0000 7f8f10a92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43148/0 (socket says 172.21.15.28:43148) 2024-09-21T05:51:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.810+0000 7f8f10a92640 1 -- 172.21.15.28:0/4170214304 learned_addr learned my addr 172.21.15.28:0/4170214304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.810+0000 7f8f11a94640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36ce7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.810+0000 7f8f11293640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c76720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 -- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 msgr2=0x55ca36c76720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c76720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 -- 172.21.15.28:0/4170214304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 msgr2=0x55ca36ce7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36ce7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 -- 172.21.15.28:0/4170214304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca36ce8190 con 0x55ca3689f390 2024-09-21T05:51:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8f10a92640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d770 tx=0x7f8efc0acce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeddd4188fdde2e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:52.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.811+0000 7f8ef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.812+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc021030 con 0x55ca3689f390 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.812+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8efc0ac8f0 con 0x55ca3689f390 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.812+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8efc0a8730 con 0x55ca3689f390 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.812+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8efc0b0070 con 0x55ca3689f390 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.813+0000 7f8f19a6b740 10 client.16770 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.813+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55ca36cef4e0 con 0x55ca3689f390 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.813+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8efc0ac8f0 con 0x55ca3689f390 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.813+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f8efc0a8930 con 0x55ca3689f390 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 10 client.16770 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 20 client.16770 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca36853da0 0x55ca36cf1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 4 client.16770 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:51:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.814+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca36bdb850 con 0x55ca36853da0 2024-09-21T05:51:52.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.818+0000 7f8f11293640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca36853da0 0x55ca36cf1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:52.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.819+0000 7f8ef4ff9640 10 client.16770 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:52.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.820+0000 7f8f11293640 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca36853da0 0x55ca36cf1960 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b75c6b8d4bdd2f3 server_cookie=516a0e08cdd803b6 in_seq=0 out_seq=0 2024-09-21T05:51:52.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.820+0000 7f8ef4ff9640 1 -- 172.21.15.28:0/4170214304 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2566 (crc 0 0 0) 0x55ca36bdb850 con 0x55ca36853da0 2024-09-21T05:51:52.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.820+0000 7f8ef4ff9640 10 client.16770 handle_command_reply: tid=0 2024-09-21T05:51:52.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.820+0000 7f8f19a6b740 1 client.16770 shutdown 2024-09-21T05:51:52.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.821+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 msgr2=0x55ca36c76d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.821+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d770 tx=0x7f8efc0acce0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 shutdown_connections 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ca36853da0 0x55ca36cf1960 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca368c0070 0x55ca36ce7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca3689f390 0x55ca36c76d00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 --2- 172.21.15.28:0/4170214304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ca368971b0 0x55ca36c76720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8ef4ff9640 0 client.16770 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.825+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 >> 172.21.15.28:0/4170214304 conn(0x55ca368b4460 msgr2=0x55ca36c68960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.828+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 shutdown_connections 2024-09-21T05:51:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.828+0000 7f8f19a6b740 1 -- 172.21.15.28:0/4170214304 wait complete. 2024-09-21T05:51:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.829+0000 7f8f19a6b740 20 client.16770 trim_cache size 0 max 16384 2024-09-21T05:51:52.831 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:51:52.831 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (113 inodes in the stack)", 2024-09-21T05:51:52.831 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:51:52.831 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "cabb93db-81c7-44a8-a5df-72df2a2e8078", 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.832 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.835 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:51:52.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 msgr2=0x7f8ef80b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 0x7f8ef80b8d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b9c0 tx=0x7f8f0802e040 comp rx=0 tx=0).stop 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 msgr2=0x7f8f0c116a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d7a0 tx=0x7f8efc00dc60 comp rx=0 tx=0).stop 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 shutdown_connections 2024-09-21T05:51:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8ef80b6850 0x7f8ef80b8d60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c10ead0 0x7f8f0c1ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c10e180 0x7f8f0c116a60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.834+0000 7f8f134f7640 1 --2- 172.21.15.28:0/3293436940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c10aee0 0x7f8f0c1164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.835+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 >> 172.21.15.28:0/3293436940 conn(0x7f8f0c070ec0 msgr2=0x7f8f0c071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.835+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 shutdown_connections 2024-09-21T05:51:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:51:52.835+0000 7f8f134f7640 1 -- 172.21.15.28:0/3293436940 wait complete. 2024-09-21T05:51:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.373+0000 7fc05da4d640 1 -- 172.21.15.28:0/975720693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058103060 msgr2=0x7fc058107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.373+0000 7fc05da4d640 1 --2- 172.21.15.28:0/975720693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058103060 0x7fc058107580 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc048009890 tx=0x7fc04802a3b0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.374+0000 7fc05da4d640 1 -- 172.21.15.28:0/975720693 shutdown_connections 2024-09-21T05:51:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.374+0000 7fc05da4d640 1 --2- 172.21.15.28:0/975720693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 0x7fc058108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.374+0000 7fc05da4d640 1 --2- 172.21.15.28:0/975720693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058103060 0x7fc058107580 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.374+0000 7fc05da4d640 1 --2- 172.21.15.28:0/975720693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058102710 0x7fc058102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.374+0000 7fc05da4d640 1 -- 172.21.15.28:0/975720693 >> 172.21.15.28:0/975720693 conn(0x7fc05807a6f0 msgr2=0x7fc05807ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.375+0000 7fc05da4d640 1 -- 172.21.15.28:0/975720693 shutdown_connections 2024-09-21T05:51:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.379+0000 7fc05da4d640 1 -- 172.21.15.28:0/975720693 wait complete. 2024-09-21T05:51:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.380+0000 7fc05da4d640 1 Processor -- start 2024-09-21T05:51:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.380+0000 7fc05da4d640 1 -- start start 2024-09-21T05:51:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.381+0000 7fc05da4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 0x7fc058116dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.381+0000 7fc05da4d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 0x7fc058111080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.381+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 0x7fc058116dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.381+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 0x7fc058116dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43168/0 (socket says 172.21.15.28:43168) 2024-09-21T05:51:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc056ffd640 1 -- 172.21.15.28:0/2877220358 learned_addr learned my addr 172.21.15.28:0/2877220358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 0x7fc0581117b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 0x7fc058111080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc058109450 con 0x7fc058102710 2024-09-21T05:51:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581092d0 con 0x7fc058103060 2024-09-21T05:51:53.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 0x7fc0581117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.382+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581095d0 con 0x7fc058107b40 2024-09-21T05:51:53.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 -- 172.21.15.28:0/2877220358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 msgr2=0x7fc0581117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 0x7fc0581117b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 -- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 msgr2=0x7fc058116dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 0x7fc058116dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0581b5450 con 0x7fc058103060 2024-09-21T05:51:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.384+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 0x7fc058111080 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc048000c00 tx=0x7fc048032cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.386+0000 7fc05ca4b640 1 -- 172.21.15.28:0/2877220358 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048037050 con 0x7fc058103060 2024-09-21T05:51:53.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.386+0000 7fc05ca4b640 1 -- 172.21.15.28:0/2877220358 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc048043070 con 0x7fc058103060 2024-09-21T05:51:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.387+0000 7fc05ca4b640 1 -- 172.21.15.28:0/2877220358 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048003a60 con 0x7fc058103060 2024-09-21T05:51:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.388+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0581b5620 con 0x7fc058103060 2024-09-21T05:51:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.388+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0581b5a90 con 0x7fc058103060 2024-09-21T05:51:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.395+0000 7fc0649bb740 1 Processor -- start 2024-09-21T05:51:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.395+0000 7fc0649bb740 1 -- start start 2024-09-21T05:51:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.395+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.395+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64febef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.396+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64fec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.396+0000 7fc0649bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc64f606f0 con 0x55fc64c461b0 2024-09-21T05:51:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.396+0000 7fc0649bb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fc64f4fb20 con 0x55fc64c254d0 2024-09-21T05:51:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.396+0000 7fc0649bb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fc64f53930 con 0x55fc64c1d2f0 2024-09-21T05:51:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.397+0000 7fc0567fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64fec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.397+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64febef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.397+0000 7fc0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.397+0000 7fc0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43184/0 (socket says 172.21.15.28:43184) 2024-09-21T05:51:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.397+0000 7fc0577fe640 1 -- 172.21.15.28:0/2682526521 learned_addr learned my addr 172.21.15.28:0/2682526521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 -- 172.21.15.28:0/2682526521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 msgr2=0x55fc64febef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64febef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 -- 172.21.15.28:0/2682526521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 msgr2=0x55fc64fec430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64fec430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 -- 172.21.15.28:0/2682526521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc64f15530 con 0x55fc64c461b0 2024-09-21T05:51:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.398+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e480 tx=0x7fc0500095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b19de5372739c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.399+0000 7fc03dffb640 1 -- 172.21.15.28:0/2682526521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050003970 con 0x55fc64c461b0 2024-09-21T05:51:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.399+0000 7fc03dffb640 1 -- 172.21.15.28:0/2682526521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05000d7d0 con 0x55fc64c461b0 2024-09-21T05:51:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.401+0000 7fc0649bb740 1 -- 172.21.15.28:0/2682526521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 msgr2=0x55fc64feb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.401+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e480 tx=0x7fc0500095c0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.402+0000 7fc0649bb740 1 -- 172.21.15.28:0/2682526521 shutdown_connections 2024-09-21T05:51:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.402+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64fec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.402+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64febef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.402+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2682526521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc64feb9b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.402+0000 7fc0649bb740 1 -- 172.21.15.28:0/2682526521 >> 172.21.15.28:0/2682526521 conn(0x55fc64c3a300 msgr2=0x55fc64fe98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.405+0000 7fc0649bb740 1 -- 172.21.15.28:0/2682526521 shutdown_connections 2024-09-21T05:51:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.405+0000 7fc0649bb740 1 -- 172.21.15.28:0/2682526521 wait complete. 2024-09-21T05:51:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.406+0000 7fc0649bb740 1 -- start start 2024-09-21T05:51:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.406+0000 7fc05ca4b640 1 -- 172.21.15.28:0/2877220358 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc048003c00 con 0x7fc058103060 2024-09-21T05:51:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.406+0000 7fc0649bb740 1 Processor -- start 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c461b0 0x55fc64ff7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc64e87ab0 con 0x55fc64c254d0 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fc64ef5730 con 0x55fc64c1d2f0 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fc64ffa1d0 con 0x55fc64c461b0 2024-09-21T05:51:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0649bb740 10 client.0 _ll_register_callbacks cb 0x55fc64f496b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc05ca4b640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 0x7fc0400b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54770/0 (socket says 172.21.15.28:54770) 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43196/0 (socket says 172.21.15.28:43196) 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0577fe640 1 -- 172.21.15.28:0/4051311197 learned_addr learned my addr 172.21.15.28:0/4051311197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 0x7fc0400b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.408+0000 7fc05ca4b640 1 -- 172.21.15.28:0/2877220358 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc048107890 con 0x7fc058103060 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 -- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c461b0 msgr2=0x55fc64ff7990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c461b0 0x55fc64ff7990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 -- 172.21.15.28:0/4051311197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 msgr2=0x55fc64ff6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 -- 172.21.15.28:0/4051311197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc64cb1790 con 0x55fc64c254d0 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc056ffd640 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005fd0 tx=0x7fc04c010b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80631c07d402d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.409+0000 7fc036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.410+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c004440 con 0x55fc64c254d0 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.410+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05000d7d0 con 0x55fc64c254d0 2024-09-21T05:51:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.410+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc04c0035f0 con 0x55fc64c254d0 2024-09-21T05:51:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.411+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc04c0037f0 con 0x55fc64c254d0 2024-09-21T05:51:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.412+0000 7fc0649bb740 10 client.16782 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.412+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fc64d2a620 con 0x55fc64c254d0 2024-09-21T05:51:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.413+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05000d7d0 con 0x55fc64c254d0 2024-09-21T05:51:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.414+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fc04c00f040 con 0x55fc64c254d0 2024-09-21T05:51:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 10 client.16782 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 20 client.16782 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64c04d80 0x55fc65009710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 4 client.16782 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.415+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc64f62640 con 0x55fc64c04d80 2024-09-21T05:51:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.416+0000 7fc0567fc640 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64c04d80 0x55fc65009710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.417+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 0x7fc0400b89c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e450 tx=0x7fc050005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.430+0000 7fc0567fc640 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64c04d80 0x55fc65009710 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f689cb1646c50707 server_cookie=abb6a9671b7f6a19 in_seq=0 out_seq=0 2024-09-21T05:51:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.431+0000 7fc036ffd640 10 client.16782 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.434+0000 7fc036ffd640 1 -- 172.21.15.28:0/4051311197 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fc64f62640 con 0x55fc64c04d80 2024-09-21T05:51:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.434+0000 7fc036ffd640 10 client.16782 handle_command_reply: tid=0 2024-09-21T05:51:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.434+0000 7fc0649bb740 1 client.16782 shutdown 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 msgr2=0x55fc64ff7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005fd0 tx=0x7fc04c010b00 comp rx=0 tx=0).stop 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 shutdown_connections 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64c04d80 0x55fc65009710 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c461b0 0x55fc64ff7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c254d0 0x55fc64ff7450 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 --2- 172.21.15.28:0/4051311197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.435+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 >> 172.21.15.28:0/4051311197 conn(0x55fc64c3a300 msgr2=0x55fc64fec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.437+0000 7fc036ffd640 0 client.16782 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.438+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 shutdown_connections 2024-09-21T05:51:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.441+0000 7fc0649bb740 1 -- 172.21.15.28:0/4051311197 wait complete. 2024-09-21T05:51:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.442+0000 7fc0649bb740 20 client.16782 trim_cache size 0 max 16384 2024-09-21T05:51:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.476+0000 7fc0649bb740 1 Processor -- start 2024-09-21T05:51:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 -- start start 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c254d0 0x55fc64ff4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc6500eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581be210 con 0x55fc64c461b0 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581bfd80 con 0x55fc64c1d2f0 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.478+0000 7fc0649bb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581be8e0 con 0x55fc64c254d0 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54784/0 (socket says 172.21.15.28:54784) 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 -- 172.21.15.28:0/2120534658 learned_addr learned my addr 172.21.15.28:0/2120534658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc6500eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 -- 172.21.15.28:0/2120534658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c254d0 msgr2=0x55fc64ff4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c254d0 0x55fc64ff4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 -- 172.21.15.28:0/2120534658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 msgr2=0x55fc6500eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc6500eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.479+0000 7fc056ffd640 1 -- 172.21.15.28:0/2120534658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc64ff9b60 con 0x55fc64c1d2f0 2024-09-21T05:51:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.480+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc04c000c00 tx=0x7fc04c0096f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537d01a919fd5216 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.481+0000 7fc03d7fa640 1 -- 172.21.15.28:0/2120534658 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c028040 con 0x55fc64c1d2f0 2024-09-21T05:51:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.481+0000 7fc03d7fa640 1 -- 172.21.15.28:0/2120534658 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc04c003e40 con 0x55fc64c1d2f0 2024-09-21T05:51:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.481+0000 7fc03d7fa640 1 -- 172.21.15.28:0/2120534658 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0118e0 con 0x55fc64c1d2f0 2024-09-21T05:51:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.482+0000 7fc0649bb740 1 -- 172.21.15.28:0/2120534658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 msgr2=0x55fc64ff40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.482+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc04c000c00 tx=0x7fc04c0096f0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.483+0000 7fc0649bb740 1 -- 172.21.15.28:0/2120534658 shutdown_connections 2024-09-21T05:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.483+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc6500eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.483+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c254d0 0x55fc64ff4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.483+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2120534658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c1d2f0 0x55fc64ff40e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.483+0000 7fc0649bb740 1 -- 172.21.15.28:0/2120534658 >> 172.21.15.28:0/2120534658 conn(0x55fc64c3a300 msgr2=0x55fc64ff5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.485+0000 7fc0649bb740 1 -- 172.21.15.28:0/2120534658 shutdown_connections 2024-09-21T05:51:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.485+0000 7fc0649bb740 1 -- 172.21.15.28:0/2120534658 wait complete. 2024-09-21T05:51:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.486+0000 7fc0649bb740 1 -- start start 2024-09-21T05:51:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.486+0000 7fc0649bb740 1 Processor -- start 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.486+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64ffc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.486+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.486+0000 7fc0649bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc650720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0649bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581bf6b0 con 0x55fc64c461b0 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0649bb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581befb0 con 0x55fc64c254d0 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0649bb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581bc2a0 con 0x55fc64c1d2f0 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0649bb740 10 client.0 _ll_register_callbacks cb 0x7fc04c0093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0567fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0567fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:54792/0 (socket says 172.21.15.28:54792) 2024-09-21T05:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc0567fc640 1 -- 172.21.15.28:0/2088672553 learned_addr learned my addr 172.21.15.28:0/2088672553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.487+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64ffc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0577fe640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc650720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0567fc640 1 -- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 msgr2=0x55fc64ffc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64ffc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0567fc640 1 -- 172.21.15.28:0/2088672553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 msgr2=0x55fc650720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc650720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.489+0000 7fc0567fc640 1 -- 172.21.15.28:0/2088672553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc65083620 con 0x55fc64c254d0 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.490+0000 7fc0567fc640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0480caa20 tx=0x7fc0480c9cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4453bff09b99ad0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.490+0000 7fc036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:51:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.493+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0480c9e00 con 0x55fc64c254d0 2024-09-21T05:51:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.494+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fc0480ccc10 con 0x55fc64c254d0 2024-09-21T05:51:53.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.494+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc04c003e40 con 0x55fc64c254d0 2024-09-21T05:51:53.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.501+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc048105430 con 0x55fc64c254d0 2024-09-21T05:51:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.501+0000 7fc0649bb740 10 client.26366 fetch_fsmap learned FSMap version 41 2024-09-21T05:51:53.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.502+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fc650847b0 con 0x55fc64c254d0 2024-09-21T05:51:53.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc04c003e40 con 0x55fc64c254d0 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fc0480c9100 con 0x55fc64c254d0 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 10 client.26366 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 10 client.26366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 20 client.26366 populate_metadata read hostname 'smithi028' 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64bda1c0 0x55fc65086ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 4 client.26366 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:51:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.504+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc64ff7dd0 con 0x55fc64bda1c0 2024-09-21T05:51:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.506+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64bda1c0 0x55fc65086ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:51:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.511+0000 7fc056ffd640 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64bda1c0 0x55fc65086ff0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd5ebc7d37da8acf server_cookie=c4e5e5c48f0b52c2 in_seq=0 out_seq=0 2024-09-21T05:51:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:53.512+0000 7fc036ffd640 10 client.26366 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:53.569 INFO:tasks.cephfs.filesystem:scrub status for tag:cabb93db-81c7-44a8-a5df-72df2a2e8078 - {'path': '/', 'tag': 'cabb93db-81c7-44a8-a5df-72df2a2e8078', 'options': 'recursive,force'} 2024-09-21T05:51:54.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:54 smithi181 ceph-mon[34819]: pgmap v471: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 36 KiB/s rd, 39 MiB/s wr, 8.68k op/s 2024-09-21T05:51:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:54 smithi028 ceph-mon[31227]: pgmap v471: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 36 KiB/s rd, 39 MiB/s wr, 8.68k op/s 2024-09-21T05:51:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:54 smithi129 ceph-mon[34885]: pgmap v471: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 36 KiB/s rd, 39 MiB/s wr, 8.68k op/s 2024-09-21T05:51:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.965+0000 7fc036ffd640 1 -- 172.21.15.28:0/2088672553 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc64ff7dd0 con 0x55fc64bda1c0 2024-09-21T05:51:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.965+0000 7fc036ffd640 10 client.26366 handle_command_reply: tid=0 2024-09-21T05:51:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 client.26366 shutdown 2024-09-21T05:51:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 msgr2=0x55fc64ffcdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0480caa20 tx=0x7fc0480c9cf0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 shutdown_connections 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc64bda1c0 0x55fc65086ff0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc64c461b0 0x55fc650720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc64c254d0 0x55fc64ffcdc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 --2- 172.21.15.28:0/2088672553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc64c1d2f0 0x55fc64ffc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 >> 172.21.15.28:0/2088672553 conn(0x55fc64c3a300 msgr2=0x55fc64fec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc036ffd640 0 client.26366 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 shutdown_connections 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.966+0000 7fc0649bb740 1 -- 172.21.15.28:0/2088672553 wait complete. 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.967+0000 7fc0649bb740 20 client.26366 trim_cache size 0 max 16384 2024-09-21T05:51:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:51:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:51:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:51:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:51:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 msgr2=0x7fc0400b89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 0x7fc0400b89c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e450 tx=0x7fc050005fd0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 msgr2=0x7fc058111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 0x7fc058111080 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc048000c00 tx=0x7fc048032cd0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 shutdown_connections 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc058107b40 0x7fc0581117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058103060 0x7fc058111080 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc0400b64b0 0x7fc0400b89c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.971+0000 7fc05da4d640 1 --2- 172.21.15.28:0/2877220358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058102710 0x7fc058116dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.972+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 >> 172.21.15.28:0/2877220358 conn(0x7fc05807a6f0 msgr2=0x7fc058105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.972+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 shutdown_connections 2024-09-21T05:51:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:51:55.972+0000 7fc05da4d640 1 -- 172.21.15.28:0/2877220358 wait complete. 2024-09-21T05:51:56.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:56 smithi181 ceph-mon[34819]: pgmap v472: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 43 KiB/s rd, 46 MiB/s wr, 10.27k op/s 2024-09-21T05:51:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:56 smithi028 ceph-mon[31227]: pgmap v472: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 43 KiB/s rd, 46 MiB/s wr, 10.27k op/s 2024-09-21T05:51:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:56 smithi129 ceph-mon[34885]: pgmap v472: 97 pgs: 97 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 43 KiB/s rd, 46 MiB/s wr, 10.27k op/s 2024-09-21T05:51:57.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:57 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:57.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:57 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:57.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:57 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:57.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:57 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:57.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:57 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:57.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:57 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:58 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:58 smithi181 ceph-mon[34819]: pgmap v473: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 12.49k op/s 2024-09-21T05:51:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:58 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:58 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:58 smithi028 ceph-mon[31227]: pgmap v473: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 12.49k op/s 2024-09-21T05:51:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:58 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:58 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:58 smithi129 ceph-mon[34885]: pgmap v473: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 12.49k op/s 2024-09-21T05:51:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:58 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:59.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:51:59 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:59.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:51:59 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:51:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:51:59 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:00 smithi181 ceph-mon[34819]: pgmap v474: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 46 KiB/s rd, 59 MiB/s wr, 13.20k op/s 2024-09-21T05:52:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:00 smithi028 ceph-mon[31227]: pgmap v474: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 46 KiB/s rd, 59 MiB/s wr, 13.20k op/s 2024-09-21T05:52:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:00 smithi129 ceph-mon[34885]: pgmap v474: 97 pgs: 97 active+clean; 15 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 46 KiB/s rd, 59 MiB/s wr, 13.20k op/s 2024-09-21T05:52:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:02 smithi181 ceph-mon[34819]: pgmap v475: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 68 KiB/s rd, 55 MiB/s wr, 12.46k op/s 2024-09-21T05:52:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:02 smithi028 ceph-mon[31227]: pgmap v475: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 68 KiB/s rd, 55 MiB/s wr, 12.46k op/s 2024-09-21T05:52:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:02 smithi129 ceph-mon[34885]: pgmap v475: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 68 KiB/s rd, 55 MiB/s wr, 12.46k op/s 2024-09-21T05:52:04.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:04 smithi181 ceph-mon[34819]: pgmap v476: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 101 KiB/s rd, 56 MiB/s wr, 12.64k op/s 2024-09-21T05:52:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:04 smithi028 ceph-mon[31227]: pgmap v476: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 101 KiB/s rd, 56 MiB/s wr, 12.64k op/s 2024-09-21T05:52:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:04 smithi129 ceph-mon[34885]: pgmap v476: 97 pgs: 97 active+clean; 15 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 101 KiB/s rd, 56 MiB/s wr, 12.64k op/s 2024-09-21T05:52:06.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:06 smithi181 ceph-mon[34819]: pgmap v477: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 102 KiB/s rd, 52 MiB/s wr, 11.66k op/s 2024-09-21T05:52:06.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:06 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:52:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:06 smithi028 ceph-mon[31227]: pgmap v477: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 102 KiB/s rd, 52 MiB/s wr, 11.66k op/s 2024-09-21T05:52:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:06 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:52:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:06 smithi129 ceph-mon[34885]: pgmap v477: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 102 KiB/s rd, 52 MiB/s wr, 11.66k op/s 2024-09-21T05:52:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:06 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:52:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:52:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:52:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:52:07.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:07 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:52:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:52:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:52:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:52:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:07 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:52:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:52:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:52:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:52:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:07 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:52:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:08 smithi181 ceph-mon[34819]: pgmap v478: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 10.04k op/s 2024-09-21T05:52:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:08 smithi028 ceph-mon[31227]: pgmap v478: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 10.04k op/s 2024-09-21T05:52:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:08 smithi129 ceph-mon[34885]: pgmap v478: 97 pgs: 97 active+clean; 16 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 10.04k op/s 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 -- 172.21.15.28:0/225002114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 msgr2=0x7f7778104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 --2- 172.21.15.28:0/225002114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f7778104bf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7768009900 tx=0x7f776802a410 comp rx=0 tx=0).stop 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 -- 172.21.15.28:0/225002114 shutdown_connections 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 --2- 172.21.15.28:0/225002114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777810dd90 0x7f7778110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 --2- 172.21.15.28:0/225002114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77781051c0 0x7f777810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 --2- 172.21.15.28:0/225002114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f7778104bf0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.129+0000 7f777f5a1640 1 -- 172.21.15.28:0/225002114 >> 172.21.15.28:0/225002114 conn(0x7f777807a740 msgr2=0x7f777807ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.130+0000 7f777f5a1640 1 -- 172.21.15.28:0/225002114 shutdown_connections 2024-09-21T05:52:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.130+0000 7f777f5a1640 1 -- 172.21.15.28:0/225002114 wait complete. 2024-09-21T05:52:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.130+0000 7f777f5a1640 1 Processor -- start 2024-09-21T05:52:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 -- start start 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77781051c0 0x7f77781a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f777810dd90 0x7f77781ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7778113a50 con 0x7f777810dd90 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77781138d0 con 0x7f77781051c0 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777f5a1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7778113bd0 con 0x7f77781047e0 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777d33d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777d33d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:47444/0 (socket says 172.21.15.28:47444) 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777d33d640 1 -- 172.21.15.28:0/188824538 learned_addr learned my addr 172.21.15.28:0/188824538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.131+0000 7f777cb3c640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77781051c0 0x7f77781a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777db3e640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f777810dd90 0x7f77781ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 -- 172.21.15.28:0/188824538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77781051c0 msgr2=0x7f77781a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77781051c0 0x7f77781a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 -- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f777810dd90 msgr2=0x7f77781ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f777810dd90 0x7f77781ac7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 -- 172.21.15.28:0/188824538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77781accf0 con 0x7f77781047e0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777d33d640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7768005940 tx=0x7f7768004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f776e7fc640 1 -- 172.21.15.28:0/188824538 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776803f030 con 0x7f77781047e0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.132+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77781acec0 con 0x7f77781047e0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.133+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77781ad270 con 0x7f77781047e0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.133+0000 7f776e7fc640 1 -- 172.21.15.28:0/188824538 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7768003aa0 con 0x7f77781047e0 2024-09-21T05:52:09.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.133+0000 7f776e7fc640 1 -- 172.21.15.28:0/188824538 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768035070 con 0x7f77781047e0 2024-09-21T05:52:09.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.136+0000 7f77865ec740 1 Processor -- start 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.137+0000 7f77865ec740 1 -- start start 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.137+0000 7f77865ec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 0x561646dd79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.137+0000 7f77865ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a092f0 0x561646dd7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.137+0000 7f77865ec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646dd8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f776e7fc640 1 -- 172.21.15.28:0/188824538 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7768042030 con 0x7f77781047e0 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777db3e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 0x561646dd79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777db3e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 0x561646dd79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:47456/0 (socket says 172.21.15.28:47456) 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777db3e640 1 -- 172.21.15.28:0/418369457 learned_addr learned my addr 172.21.15.28:0/418369457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:09.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561646d4c6f0 con 0x561646a092f0 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561646d3bb20 con 0x561646a114d0 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561646d3f930 con 0x561646a321b0 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646dd8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 -- 172.21.15.28:0/418369457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 msgr2=0x561646dd79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 0x561646dd79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 -- 172.21.15.28:0/418369457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a092f0 msgr2=0x561646dd7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a092f0 0x561646dd7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.138+0000 7f777cb3c640 1 -- 172.21.15.28:0/418369457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561646d01530 con 0x561646a114d0 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.139+0000 7f777cb3c640 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646dd8430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f777806c9c0 tx=0x7f7774011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=443022d757bf63b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.139+0000 7f77637fe640 1 -- 172.21.15.28:0/418369457 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774009d70 con 0x561646a114d0 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.139+0000 7f77637fe640 1 -- 172.21.15.28:0/418369457 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7774005020 con 0x561646a114d0 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.140+0000 7f776e7fc640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 0x7f77640b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.141+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 msgr2=0x561646dd8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.141+0000 7f77865ec740 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646dd8430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f777806c9c0 tx=0x7f7774011810 comp rx=0 tx=0).stop 2024-09-21T05:52:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.141+0000 7f777db3e640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 0x7f77640b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.141+0000 7f776e7fc640 1 -- 172.21.15.28:0/188824538 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7768107af0 con 0x7f77781047e0 2024-09-21T05:52:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f777db3e640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 0x7f77640b8d60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f777000b600 tx=0x7f777000e040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 shutdown_connections 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646dd8430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a092f0 0x561646dd7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 --2- 172.21.15.28:0/418369457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a321b0 0x561646dd79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 >> 172.21.15.28:0/418369457 conn(0x561646a26300 msgr2=0x561646dd58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 shutdown_connections 2024-09-21T05:52:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.142+0000 7f77865ec740 1 -- 172.21.15.28:0/418369457 wait complete. 2024-09-21T05:52:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.143+0000 7f77865ec740 1 -- start start 2024-09-21T05:52:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.143+0000 7f77865ec740 1 Processor -- start 2024-09-21T05:52:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646de3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a114d0 0x561646de3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 0x561646de3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561646c73ab0 con 0x561646a114d0 2024-09-21T05:52:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561646ce1730 con 0x561646a321b0 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561646de61d0 con 0x561646a092f0 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f77865ec740 10 client.0 _ll_register_callbacks cb 0x561646d356b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f777cb3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a114d0 0x561646de3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f777db3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 0x561646de3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f777db3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 0x561646de3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:55420/0 (socket says 172.21.15.28:55420) 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f777db3e640 1 -- 172.21.15.28:0/1282994313 learned_addr learned my addr 172.21.15.28:0/1282994313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.144+0000 7f777d33d640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646de3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.145+0000 7f777d33d640 1 -- 172.21.15.28:0/1282994313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 msgr2=0x561646de3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.145+0000 7f777d33d640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 0x561646de3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.145+0000 7f777d33d640 1 -- 172.21.15.28:0/1282994313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a114d0 msgr2=0x561646de3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.145+0000 7f777d33d640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a114d0 0x561646de3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.145+0000 7f777d33d640 1 -- 172.21.15.28:0/1282994313 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561646a9d790 con 0x561646a092f0 2024-09-21T05:52:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.146+0000 7f777d33d640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646de3000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7768105200 tx=0x7f77680c9860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c1c5ba9e3563a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.147+0000 7f7760ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.147+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768049070 con 0x561646a092f0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.148+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7774005020 con 0x561646a092f0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.148+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7768105480 con 0x561646a092f0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.148+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7768105680 con 0x561646a092f0 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.148+0000 7f77865ec740 10 client.26392 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.148+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561646b16620 con 0x561646a092f0 2024-09-21T05:52:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7774005020 con 0x561646a092f0 2024-09-21T05:52:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f77680d6e90 con 0x561646a092f0 2024-09-21T05:52:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 10 client.26392 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 10 client.26392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 20 client.26392 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469f0d80 0x561646e06200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 4 client.26392 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.149+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561646d4e640 con 0x5616469f0d80 2024-09-21T05:52:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.150+0000 7f777cb3c640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469f0d80 0x561646e06200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.153+0000 7f777cb3c640 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469f0d80 0x561646e06200 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60dc48f7390527ff server_cookie=7e79d72de8568746 in_seq=0 out_seq=0 2024-09-21T05:52:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.153+0000 7f7760ff9640 10 client.26392 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f7760ff9640 1 -- 172.21.15.28:0/1282994313 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561646d4e640 con 0x5616469f0d80 2024-09-21T05:52:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f7760ff9640 10 client.26392 handle_command_reply: tid=0 2024-09-21T05:52:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 client.26392 shutdown 2024-09-21T05:52:09.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 msgr2=0x561646de3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646de3000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7768105200 tx=0x7f77680c9860 comp rx=0 tx=0).stop 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 shutdown_connections 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469f0d80 0x561646e06200 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a321b0 0x561646de3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a114d0 0x561646de3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 --2- 172.21.15.28:0/1282994313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646de3000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 >> 172.21.15.28:0/1282994313 conn(0x561646a26300 msgr2=0x561646dd89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f7760ff9640 0 client.26392 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 shutdown_connections 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.155+0000 7f77865ec740 1 -- 172.21.15.28:0/1282994313 wait complete. 2024-09-21T05:52:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.156+0000 7f77865ec740 20 client.26392 trim_cache size 0 max 16384 2024-09-21T05:52:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 Processor -- start 2024-09-21T05:52:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 -- start start 2024-09-21T05:52:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646dd5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646de01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.178+0000 7f77865ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00e820 con 0x561646a321b0 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f77865ec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f775c010390 con 0x561646a114d0 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f77865ec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00eef0 con 0x561646a092f0 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58668/0 (socket says 172.21.15.28:58668) 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777cb3c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646de01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 -- 172.21.15.28:0/3586911437 learned_addr learned my addr 172.21.15.28:0/3586911437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777d33d640 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646dd5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 -- 172.21.15.28:0/3586911437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 msgr2=0x561646dd5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646dd5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 -- 172.21.15.28:0/3586911437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 msgr2=0x561646de01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646de01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777db3e640 1 -- 172.21.15.28:0/3586911437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561646e06740 con 0x561646a321b0 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.179+0000 7f777cb3c640 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646de01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:52:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.180+0000 7f777db3e640 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7770009fd0 tx=0x7f7770013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e5c8f9ac4678e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.180+0000 7f7762ffd640 1 -- 172.21.15.28:0/3586911437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770027070 con 0x561646a321b0 2024-09-21T05:52:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.180+0000 7f7762ffd640 1 -- 172.21.15.28:0/3586911437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7770018b80 con 0x561646a321b0 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.180+0000 7f7762ffd640 1 -- 172.21.15.28:0/3586911437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770023670 con 0x561646a321b0 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.181+0000 7f77865ec740 1 -- 172.21.15.28:0/3586911437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 msgr2=0x561646de0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.181+0000 7f77865ec740 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7770009fd0 tx=0x7f7770013000 comp rx=0 tx=0).stop 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.181+0000 7f77865ec740 1 -- 172.21.15.28:0/3586911437 shutdown_connections 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646de0710 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a114d0 0x561646de01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 --2- 172.21.15.28:0/3586911437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a092f0 0x561646dd5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 -- 172.21.15.28:0/3586911437 >> 172.21.15.28:0/3586911437 conn(0x561646a26300 msgr2=0x561646dd50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 -- 172.21.15.28:0/3586911437 shutdown_connections 2024-09-21T05:52:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.182+0000 7f77865ec740 1 -- 172.21.15.28:0/3586911437 wait complete. 2024-09-21T05:52:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.183+0000 7f77865ec740 1 -- start start 2024-09-21T05:52:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.183+0000 7f77865ec740 1 Processor -- start 2024-09-21T05:52:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a092f0 0x561646de8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a114d0 0x561646de9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00fcc0 con 0x561646a321b0 2024-09-21T05:52:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00f5c0 con 0x561646a092f0 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00c800 con 0x561646a114d0 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f77865ec740 10 client.0 _ll_register_callbacks cb 0x7f7768005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f777db3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f777db3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58676/0 (socket says 172.21.15.28:58676) 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.184+0000 7f777db3e640 1 -- 172.21.15.28:0/1659185948 learned_addr learned my addr 172.21.15.28:0/1659185948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 -- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a114d0 msgr2=0x561646de9380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a114d0 0x561646de9380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 -- 172.21.15.28:0/1659185948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a092f0 msgr2=0x561646de8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a092f0 0x561646de8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 -- 172.21.15.28:0/1659185948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561646e69230 con 0x561646a321b0 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.185+0000 7f777db3e640 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7770009f20 tx=0x7f77700076f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8448e23288d38d55 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f7760ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770027040 con 0x561646a321b0 2024-09-21T05:52:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7770022a40 con 0x561646a321b0 2024-09-21T05:52:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7770018b80 con 0x561646a321b0 2024-09-21T05:52:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f777002c450 con 0x561646a321b0 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f77865ec740 10 client.16800 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.186+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561646e70500 con 0x561646a321b0 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f777002caa0 con 0x561646a321b0 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f77700406e0 con 0x561646a321b0 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 10 client.16800 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 20 client.16800 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469c61c0 0x561646e72b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 4 client.16800 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:52:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.187+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7770018b80 con 0x5616469c61c0 2024-09-21T05:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.188+0000 7f777d33d640 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469c61c0 0x561646e72b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.189+0000 7f7760ff9640 10 client.16800 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.189+0000 7f777d33d640 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469c61c0 0x561646e72b90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de48cbbd815051c6 server_cookie=f5185c75d05ee9a7 in_seq=0 out_seq=0 2024-09-21T05:52:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.216+0000 7f7760ff9640 1 -- 172.21.15.28:0/1659185948 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7770018b80 con 0x5616469c61c0 2024-09-21T05:52:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.216+0000 7f7760ff9640 10 client.16800 handle_command_reply: tid=0 2024-09-21T05:52:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.216+0000 7f77865ec740 1 client.16800 shutdown 2024-09-21T05:52:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.216+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 msgr2=0x561646e68b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7770009f20 tx=0x7f77700076f0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 shutdown_connections 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5616469c61c0 0x561646e72b90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f7760ff9640 0 client.16800 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561646a321b0 0x561646e68b90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561646a114d0 0x561646de9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 --2- 172.21.15.28:0/1659185948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561646a092f0 0x561646de8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 >> 172.21.15.28:0/1659185948 conn(0x561646a26300 msgr2=0x561646de44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.217+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 shutdown_connections 2024-09-21T05:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.218+0000 7f77865ec740 1 -- 172.21.15.28:0/1659185948 wait complete. 2024-09-21T05:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.218+0000 7f77865ec740 20 client.16800 trim_cache size 0 max 16384 2024-09-21T05:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 msgr2=0x7f77640b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 0x7f77640b8d60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f777000b600 tx=0x7f777000e040 comp rx=0 tx=0).stop 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 msgr2=0x7f77781a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7768005940 tx=0x7f7768004270 comp rx=0 tx=0).stop 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 shutdown_connections 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77640b6850 0x7f77640b8d60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f777810dd90 0x7f77781ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77781051c0 0x7f77781a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 --2- 172.21.15.28:0/188824538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77781047e0 0x7f77781a1730 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 >> 172.21.15.28:0/188824538 conn(0x7f777807a740 msgr2=0x7f7778100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 shutdown_connections 2024-09-21T05:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:09.219+0000 7f777f5a1640 1 -- 172.21.15.28:0/188824538 wait complete. 2024-09-21T05:52:10.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:10 smithi181 ceph-mon[34819]: pgmap v479: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 101 KiB/s rd, 30 MiB/s wr, 6.76k op/s 2024-09-21T05:52:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:10 smithi028 ceph-mon[31227]: pgmap v479: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 101 KiB/s rd, 30 MiB/s wr, 6.76k op/s 2024-09-21T05:52:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:10 smithi129 ceph-mon[34885]: pgmap v479: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 101 KiB/s rd, 30 MiB/s wr, 6.76k op/s 2024-09-21T05:52:12.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:12 smithi181 ceph-mon[34819]: pgmap v480: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 89 KiB/s rd, 20 MiB/s wr, 4.63k op/s 2024-09-21T05:52:12.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:12 smithi028 ceph-mon[31227]: pgmap v480: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 89 KiB/s rd, 20 MiB/s wr, 4.63k op/s 2024-09-21T05:52:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:12 smithi129 ceph-mon[34885]: pgmap v480: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 89 KiB/s rd, 20 MiB/s wr, 4.63k op/s 2024-09-21T05:52:14.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:14 smithi181 ceph-mon[34819]: pgmap v481: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 83 KiB/s rd, 17 MiB/s wr, 3.96k op/s 2024-09-21T05:52:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:14 smithi028 ceph-mon[31227]: pgmap v481: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 83 KiB/s rd, 17 MiB/s wr, 3.96k op/s 2024-09-21T05:52:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:14 smithi129 ceph-mon[34885]: pgmap v481: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 83 KiB/s rd, 17 MiB/s wr, 3.96k op/s 2024-09-21T05:52:16.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:16 smithi181 ceph-mon[34819]: pgmap v482: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-09-21T05:52:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:16 smithi028 ceph-mon[31227]: pgmap v482: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-09-21T05:52:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:16 smithi129 ceph-mon[34885]: pgmap v482: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-09-21T05:52:18.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:18 smithi181 ceph-mon[34819]: pgmap v483: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-09-21T05:52:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:18 smithi028 ceph-mon[31227]: pgmap v483: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-09-21T05:52:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:18 smithi129 ceph-mon[34885]: pgmap v483: 97 pgs: 97 active+clean; 16 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-09-21T05:52:20.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:20 smithi181 ceph-mon[34819]: pgmap v484: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 46 KiB/s rd, 23 MiB/s wr, 5.43k op/s 2024-09-21T05:52:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:20 smithi028 ceph-mon[31227]: pgmap v484: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 46 KiB/s rd, 23 MiB/s wr, 5.43k op/s 2024-09-21T05:52:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:20 smithi129 ceph-mon[34885]: pgmap v484: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 46 KiB/s rd, 23 MiB/s wr, 5.43k op/s 2024-09-21T05:52:22.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 -- 172.21.15.28:0/3442905946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 msgr2=0x7f2184103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 --2- 172.21.15.28:0/3442905946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184103ee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f217c01acb0 tx=0x7f217c03b900 comp rx=0 tx=0).stop 2024-09-21T05:52:22.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 -- 172.21.15.28:0/3442905946 shutdown_connections 2024-09-21T05:52:22.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 --2- 172.21.15.28:0/3442905946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2184109d50 0x7f2184110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 --2- 172.21.15.28:0/3442905946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2184109400 0x7f2184109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 --2- 172.21.15.28:0/3442905946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184103ee0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.395+0000 7f218b707640 1 -- 172.21.15.28:0/3442905946 >> 172.21.15.28:0/3442905946 conn(0x7f21840ff510 msgr2=0x7f2184101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.397+0000 7f218b707640 1 -- 172.21.15.28:0/3442905946 shutdown_connections 2024-09-21T05:52:22.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.398+0000 7f218b707640 1 -- 172.21.15.28:0/3442905946 wait complete. 2024-09-21T05:52:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 Processor -- start 2024-09-21T05:52:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 -- start start 2024-09-21T05:52:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2184109400 0x7f2184071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2184109d50 0x7f2184071750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f218406f730 con 0x7f2184109d50 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f218406f5b0 con 0x7f2184109400 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f218b707640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f218406f8b0 con 0x7f2184103ad0 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.399+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58674/0 (socket says 172.21.15.28:58674) 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 -- 172.21.15.28:0/1271534490 learned_addr learned my addr 172.21.15.28:0/1271534490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 -- 172.21.15.28:0/1271534490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2184109400 msgr2=0x7f2184071210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2184109400 0x7f2184071210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 -- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2184109d50 msgr2=0x7f2184071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2184109d50 0x7f2184071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 -- 172.21.15.28:0/1271534490 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2184071e80 con 0x7f2184103ad0 2024-09-21T05:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.400+0000 7f21894a3640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f217c03bef0 tx=0x7f217c003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.401+0000 7f217a7fc640 1 -- 172.21.15.28:0/1271534490 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c045070 con 0x7f2184103ad0 2024-09-21T05:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.401+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2184195ee0 con 0x7f2184103ad0 2024-09-21T05:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.401+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2184196450 con 0x7f2184103ad0 2024-09-21T05:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.401+0000 7f217a7fc640 1 -- 172.21.15.28:0/1271534490 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f217c003e10 con 0x7f2184103ad0 2024-09-21T05:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.401+0000 7f217a7fc640 1 -- 172.21.15.28:0/1271534490 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c0046b0 con 0x7f2184103ad0 2024-09-21T05:52:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.407+0000 7f217a7fc640 1 -- 172.21.15.28:0/1271534490 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f217c048020 con 0x7f2184103ad0 2024-09-21T05:52:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.408+0000 7f2192672740 1 Processor -- start 2024-09-21T05:52:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 -- start start 2024-09-21T05:52:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b019b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 0x55b6e3b01ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f217a7fc640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 0x7f21700b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b02430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6e3a766f0 con 0x55b6e37332f0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b6e3a65b20 con 0x55b6e375c1b0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2192672740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b6e3a69930 con 0x55b6e373b4d0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f217a7fc640 1 -- 172.21.15.28:0/1271534490 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f217c119240 con 0x7f2184103ad0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f21894a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 0x55b6e3b01ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f21894a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 0x55b6e3b01ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60222/0 (socket says 172.21.15.28:60222) 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f21894a3640 1 -- 172.21.15.28:0/162853688 learned_addr learned my addr 172.21.15.28:0/162853688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2189ca4640 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b02430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.409+0000 7f2189ca4640 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 0x7f21700b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 -- 172.21.15.28:0/162853688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 msgr2=0x55b6e3b019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 -- 172.21.15.28:0/162853688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 msgr2=0x55b6e3b01ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 0x55b6e3b01ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 -- 172.21.15.28:0/162853688 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e3a2b530 con 0x55b6e373b4d0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2189ca4640 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b02430 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2180001e70 tx=0x7f2180009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95c0102b91cd5439 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 0x7f21700b8d50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f21841067c0 tx=0x7f2174006bd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.410+0000 7f216f7fe640 1 -- 172.21.15.28:0/162853688 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180012040 con 0x55b6e373b4d0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.411+0000 7f216f7fe640 1 -- 172.21.15.28:0/162853688 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2180008e30 con 0x55b6e373b4d0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.411+0000 7f216f7fe640 1 -- 172.21.15.28:0/162853688 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180004a30 con 0x55b6e373b4d0 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 -- 172.21.15.28:0/162853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 msgr2=0x55b6e3b02430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b02430 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2180001e70 tx=0x7f2180009da0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 -- 172.21.15.28:0/162853688 shutdown_connections 2024-09-21T05:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b02430 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e37332f0 0x55b6e3b01ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 --2- 172.21.15.28:0/162853688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.412+0000 7f2192672740 1 -- 172.21.15.28:0/162853688 >> 172.21.15.28:0/162853688 conn(0x55b6e3750300 msgr2=0x55b6e3aff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.413+0000 7f2192672740 1 -- 172.21.15.28:0/162853688 shutdown_connections 2024-09-21T05:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.413+0000 7f2192672740 1 -- 172.21.15.28:0/162853688 wait complete. 2024-09-21T05:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.414+0000 7f2192672740 1 -- start start 2024-09-21T05:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.414+0000 7f2192672740 1 Processor -- start 2024-09-21T05:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f2192672740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f2192672740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58690/0 (socket says 172.21.15.28:58690) 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f21894a3640 1 -- 172.21.15.28:0/1843093921 learned_addr learned my addr 172.21.15.28:0/1843093921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.415+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b19420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6e399dab0 con 0x55b6e373b4d0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b6e3a0b730 con 0x55b6e375c1b0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2189ca4640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b19420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b6e3b100d0 con 0x55b6e37332f0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2192672740 10 client.0 _ll_register_callbacks cb 0x55b6e3a5f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f21894a3640 1 -- 172.21.15.28:0/1843093921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 msgr2=0x55b6e3b19420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f21894a3640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b19420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f21894a3640 1 -- 172.21.15.28:0/1843093921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 msgr2=0x55b6e3b0d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f21894a3640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.416+0000 7f21894a3640 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e37c7790 con 0x55b6e37332f0 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f21894a3640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f217c01a440 tx=0x7f217c0dbfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2da01f602057e3d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f216cff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f216cff9640 1 -- 172.21.15.28:0/1843093921 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c0e4c20 con 0x55b6e37332f0 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f216cff9640 1 -- 172.21.15.28:0/1843093921 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f217c004040 con 0x55b6e37332f0 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:52:22.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.417+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2180008e30 con 0x55b6e37332f0 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.418+0000 7f216cff9640 1 -- 172.21.15.28:0/1843093921 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f217c0e9b60 con 0x55b6e37332f0 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.418+0000 7f2192672740 10 client.26404 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.418+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=41}) -- 0x55b6e3840620 con 0x55b6e37332f0 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.418+0000 7f216cff9640 1 -- 172.21.15.28:0/1843093921 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f217c116bd0 con 0x55b6e37332f0 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 10 client.26404 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:22.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 20 client.26404 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:22.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e371ad80 0x55b6e3b1f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 4 client.26404 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:22.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.419+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2180008e30 con 0x55b6e371ad80 2024-09-21T05:52:22.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.421+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e371ad80 0x55b6e3b1f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.422+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e371ad80 0x55b6e3b1f040 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27c51713d9c8523a server_cookie=c543d7d3bd10f878 in_seq=0 out_seq=0 2024-09-21T05:52:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.422+0000 7f216cff9640 10 client.26404 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.423+0000 7f216cff9640 1 -- 172.21.15.28:0/1843093921 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f2180008e30 con 0x55b6e371ad80 2024-09-21T05:52:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.423+0000 7f216cff9640 10 client.26404 handle_command_reply: tid=0 2024-09-21T05:52:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.423+0000 7f2192672740 1 client.26404 shutdown 2024-09-21T05:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.430+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 msgr2=0x55b6e3b0d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.430+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f217c01a440 tx=0x7f217c0dbfe0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 shutdown_connections 2024-09-21T05:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e371ad80 0x55b6e3b1f040 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b19420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 --2- 172.21.15.28:0/1843093921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d0e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f216cff9640 0 client.26404 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 >> 172.21.15.28:0/1843093921 conn(0x55b6e3750300 msgr2=0x55b6e3b029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.432+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 shutdown_connections 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.433+0000 7f2192672740 1 -- 172.21.15.28:0/1843093921 wait complete. 2024-09-21T05:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.434+0000 7f2192672740 20 client.26404 trim_cache size 0 max 16384 2024-09-21T05:52:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.465+0000 7f2192672740 1 Processor -- start 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.465+0000 7f2192672740 1 -- start start 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 0x55b6e3b030f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216800e790 con 0x55b6e375c1b0 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2168010300 con 0x55b6e37332f0 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.466+0000 7f2192672740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f216800ee60 con 0x55b6e373b4d0 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.467+0000 7f2188ca2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.467+0000 7f2188ca2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58698/0 (socket says 172.21.15.28:58698) 2024-09-21T05:52:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.467+0000 7f2188ca2640 1 -- 172.21.15.28:0/1505930583 learned_addr learned my addr 172.21.15.28:0/1505930583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.467+0000 7f21894a3640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.468+0000 7f2189ca4640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 0x55b6e3b030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.468+0000 7f2188ca2640 1 -- 172.21.15.28:0/1505930583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 msgr2=0x55b6e3b0d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f2188ca2640 1 -- 172.21.15.28:0/1505930583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 msgr2=0x55b6e3b030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 0x55b6e3b030f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f2188ca2640 1 -- 172.21.15.28:0/1505930583 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e3b0dca0 con 0x55b6e373b4d0 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f21894a3640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:52:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.469+0000 7f2188ca2640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55b6e3b1f7b0 tx=0x7f2174012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38d1bb45bf75b1d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.470+0000 7f216effd640 1 -- 172.21.15.28:0/1505930583 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174012b20 con 0x55b6e373b4d0 2024-09-21T05:52:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.470+0000 7f216effd640 1 -- 172.21.15.28:0/1505930583 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f217402b070 con 0x55b6e373b4d0 2024-09-21T05:52:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.470+0000 7f216effd640 1 -- 172.21.15.28:0/1505930583 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217401f650 con 0x55b6e373b4d0 2024-09-21T05:52:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.470+0000 7f2189ca4640 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 0x55b6e3b030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 -- 172.21.15.28:0/1505930583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 msgr2=0x55b6e3b0efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55b6e3b1f7b0 tx=0x7f2174012000 comp rx=0 tx=0).stop 2024-09-21T05:52:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 -- 172.21.15.28:0/1505930583 shutdown_connections 2024-09-21T05:52:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e375c1b0 0x55b6e3b030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e373b4d0 0x55b6e3b0efc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 --2- 172.21.15.28:0/1505930583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e37332f0 0x55b6e3b0d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 -- 172.21.15.28:0/1505930583 >> 172.21.15.28:0/1505930583 conn(0x55b6e3750300 msgr2=0x55b6e3b029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 -- 172.21.15.28:0/1505930583 shutdown_connections 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.471+0000 7f2192672740 1 -- 172.21.15.28:0/1505930583 wait complete. 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.472+0000 7f2192672740 1 -- start start 2024-09-21T05:52:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.472+0000 7f2192672740 1 Processor -- start 2024-09-21T05:52:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b11320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b81ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216800fc30 con 0x55b6e373b4d0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f216800f530 con 0x55b6e375c1b0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f216800c770 con 0x55b6e37332f0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2192672740 10 client.0 _ll_register_callbacks cb 0x7f217c0dc5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f21894a3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58704/0 (socket says 172.21.15.28:58704) 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f21894a3640 1 -- 172.21.15.28:0/802886008 learned_addr learned my addr 172.21.15.28:0/802886008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2189ca4640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b81ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f2188ca2640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b11320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.473+0000 7f21894a3640 1 -- 172.21.15.28:0/802886008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 msgr2=0x55b6e3b81ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f21894a3640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b81ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f21894a3640 1 -- 172.21.15.28:0/802886008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 msgr2=0x55b6e3b11320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f21894a3640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b11320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f21894a3640 1 -- 172.21.15.28:0/802886008 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e3b82180 con 0x55b6e37332f0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f21894a3640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f217c0dd3b0 tx=0x7f217c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d9b51044962eb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f216cff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c0e4c20 con 0x55b6e37332f0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.474+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f217c0df5b0 con 0x55b6e37332f0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f217402b070 con 0x55b6e37332f0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f217c0dba90 con 0x55b6e37332f0 2024-09-21T05:52:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f2192672740 10 client.26410 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55b6e3b894c0 con 0x55b6e37332f0 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f217402b070 con 0x55b6e37332f0 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.475+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f217c0f7060 con 0x55b6e37332f0 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 10 client.26410 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 10 client.26410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 20 client.26410 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e36f01c0 0x55b6e3b8bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 4 client.26410 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6e3a78640 con 0x55b6e36f01c0 2024-09-21T05:52:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.476+0000 7f2188ca2640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e36f01c0 0x55b6e3b8bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.478+0000 7f2188ca2640 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e36f01c0 0x55b6e3b8bad0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b16fb15cf20e9f14 server_cookie=9c58dfc32c3424ee in_seq=0 out_seq=0 2024-09-21T05:52:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.478+0000 7f216cff9640 10 client.26410 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.515+0000 7f216cff9640 1 -- 172.21.15.28:0/802886008 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b6e3a78640 con 0x55b6e36f01c0 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.515+0000 7f216cff9640 10 client.26410 handle_command_reply: tid=0 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.515+0000 7f2192672740 1 client.26410 shutdown 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 msgr2=0x55b6e3b10d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f217c0dd3b0 tx=0x7f217c000f40 comp rx=0 tx=0).stop 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 shutdown_connections 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b6e36f01c0 0x55b6e3b8bad0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b6e375c1b0 0x55b6e3b81ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6e373b4d0 0x55b6e3b11320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 --2- 172.21.15.28:0/802886008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b6e37332f0 0x55b6e3b10d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 >> 172.21.15.28:0/802886008 conn(0x55b6e3750300 msgr2=0x55b6e3b0ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f216cff9640 0 client.26410 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.516+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 shutdown_connections 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.517+0000 7f2192672740 1 -- 172.21.15.28:0/802886008 wait complete. 2024-09-21T05:52:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.517+0000 7f2192672740 20 client.26410 trim_cache size 0 max 16384 2024-09-21T05:52:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:52:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:52:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:52:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:52:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.520+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 msgr2=0x7f21700b8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.520+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 0x7f21700b8d50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f21841067c0 tx=0x7f2174006bd0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.521+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 msgr2=0x7f2184076f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.521+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f217c03bef0 tx=0x7f217c003c20 comp rx=0 tx=0).stop 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 shutdown_connections 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f21700b6840 0x7f21700b8d50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2184109d50 0x7f2184071750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2184109400 0x7f2184071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 --2- 172.21.15.28:0/1271534490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2184103ad0 0x7f2184076f60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.522+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 >> 172.21.15.28:0/1271534490 conn(0x7f21840ff510 msgr2=0x7f2184101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.523+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 shutdown_connections 2024-09-21T05:52:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:22.523+0000 7f218b707640 1 -- 172.21.15.28:0/1271534490 wait complete. 2024-09-21T05:52:22.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:22 smithi181 ceph-mon[34819]: pgmap v485: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 6.24k op/s 2024-09-21T05:52:22.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:22 smithi028 ceph-mon[31227]: pgmap v485: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 6.24k op/s 2024-09-21T05:52:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:22 smithi129 ceph-mon[34885]: pgmap v485: 97 pgs: 97 active+clean; 16 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 6.24k op/s 2024-09-21T05:52:23.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:52:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:24.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:24 smithi028 ceph-mon[31227]: pgmap v486: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 53 KiB/s rd, 32 MiB/s wr, 7.47k op/s 2024-09-21T05:52:24.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:24 smithi181 ceph-mon[34819]: pgmap v486: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 53 KiB/s rd, 32 MiB/s wr, 7.47k op/s 2024-09-21T05:52:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:24 smithi129 ceph-mon[34885]: pgmap v486: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 53 KiB/s rd, 32 MiB/s wr, 7.47k op/s 2024-09-21T05:52:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.623+0000 7f824eb77640 1 -- 172.21.15.28:0/505266268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 msgr2=0x7f82481029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.623+0000 7f824eb77640 1 --2- 172.21.15.28:0/505266268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f82481029c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f824401a910 tx=0x7f824403b430 comp rx=0 tx=0).stop 2024-09-21T05:52:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.624+0000 7f824eb77640 1 -- 172.21.15.28:0/505266268 shutdown_connections 2024-09-21T05:52:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.624+0000 7f824eb77640 1 --2- 172.21.15.28:0/505266268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82481079f0 0x7f8248107ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.624+0000 7f824eb77640 1 --2- 172.21.15.28:0/505266268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248102f00 0x7f8248107430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.624+0000 7f824eb77640 1 --2- 172.21.15.28:0/505266268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f82481029c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.624+0000 7f824eb77640 1 -- 172.21.15.28:0/505266268 >> 172.21.15.28:0/505266268 conn(0x7f82480fe2c0 msgr2=0x7f8248100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.625+0000 7f824eb77640 1 -- 172.21.15.28:0/505266268 shutdown_connections 2024-09-21T05:52:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.626+0000 7f824eb77640 1 -- 172.21.15.28:0/505266268 wait complete. 2024-09-21T05:52:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.627+0000 7f824eb77640 1 Processor -- start 2024-09-21T05:52:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.627+0000 7f824eb77640 1 -- start start 2024-09-21T05:52:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f8248116ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8248102f00 0x7f8248112dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f824810b290 con 0x7f82481079f0 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f824810b110 con 0x7f8248102f00 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824eb77640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f824810b410 con 0x7f82481025b0 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824e376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824db75640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f8248116ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824e376640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824e376640 1 -- 172.21.15.28:0/2462508591 learned_addr learned my addr 172.21.15.28:0/2462508591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.628+0000 7f824d374640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8248102f00 0x7f8248112dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 -- 172.21.15.28:0/2462508591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 msgr2=0x7f8248116ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f8248116ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 -- 172.21.15.28:0/2462508591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8248102f00 msgr2=0x7f8248112dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8248102f00 0x7f8248112dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 -- 172.21.15.28:0/2462508591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8248113b80 con 0x7f82481079f0 2024-09-21T05:52:24.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f824e376640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f824000efd0 tx=0x7f824000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.629+0000 7f823effd640 1 -- 172.21.15.28:0/2462508591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240003b70 con 0x7f82481079f0 2024-09-21T05:52:24.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.630+0000 7f823effd640 1 -- 172.21.15.28:0/2462508591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8240003d10 con 0x7f82481079f0 2024-09-21T05:52:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.630+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82481a4470 con 0x7f82481079f0 2024-09-21T05:52:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.630+0000 7f823effd640 1 -- 172.21.15.28:0/2462508591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240010820 con 0x7f82481079f0 2024-09-21T05:52:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.630+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a49b0 con 0x7f82481079f0 2024-09-21T05:52:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.635+0000 7f825634d740 1 Processor -- start 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.636+0000 7f823effd640 1 -- 172.21.15.28:0/2462508591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f82400109c0 con 0x7f82481079f0 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 -- start start 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 0x563cd2dc5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dc6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cd2d3be50 con 0x563cd29f51b0 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563cd2d2b280 con 0x563cd2a1e070 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f825634d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563cd2d2f090 con 0x563cd29fd390 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 0x563cd2dc5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.637+0000 7f824db75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60280/0 (socket says 172.21.15.28:60280) 2024-09-21T05:52:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 0x563cd2dc5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60624/0 (socket says 172.21.15.28:60624) 2024-09-21T05:52:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 -- 172.21.15.28:0/646702236 learned_addr learned my addr 172.21.15.28:0/646702236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824d374640 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dc6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f823effd640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 0x7f82340b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 -- 172.21.15.28:0/646702236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 msgr2=0x563cd2dc6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dc6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 -- 172.21.15.28:0/646702236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 msgr2=0x563cd2dc5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 0x563cd2dc5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824db75640 1 -- 172.21.15.28:0/646702236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cd2a89650 con 0x563cd29f51b0 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f824e376640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 0x7f82340b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.638+0000 7f823effd640 1 -- 172.21.15.28:0/2462508591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f82400e7ca0 con 0x7f82481079f0 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.639+0000 7f824db75640 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f824403b950 tx=0x7f82440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11c3422e0d8621a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.639+0000 7f8233fff640 1 -- 172.21.15.28:0/646702236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244046030 con 0x563cd29f51b0 2024-09-21T05:52:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.639+0000 7f824e376640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 0x7f82340b8d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8240016000 tx=0x7f8240013000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.639+0000 7f8233fff640 1 -- 172.21.15.28:0/646702236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8244004030 con 0x563cd29f51b0 2024-09-21T05:52:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.640+0000 7f8233fff640 1 -- 172.21.15.28:0/646702236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82440054e0 con 0x563cd29f51b0 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 -- 172.21.15.28:0/646702236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 msgr2=0x563cd2dc5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f824403b950 tx=0x7f82440046c0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 -- 172.21.15.28:0/646702236 shutdown_connections 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dc6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29f51b0 0x563cd2dc5d50 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 --2- 172.21.15.28:0/646702236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2a1e070 0x563cd2dc5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.641+0000 7f825634d740 1 -- 172.21.15.28:0/646702236 >> 172.21.15.28:0/646702236 conn(0x563cd2a12460 msgr2=0x563cd2dc3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.642+0000 7f825634d740 1 -- 172.21.15.28:0/646702236 shutdown_connections 2024-09-21T05:52:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.642+0000 7f825634d740 1 -- 172.21.15.28:0/646702236 wait complete. 2024-09-21T05:52:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.642+0000 7f825634d740 1 -- start start 2024-09-21T05:52:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.642+0000 7f825634d740 1 Processor -- start 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dd1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2ddd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cd2b8c700 con 0x563cd29fd390 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563cd2cc6110 con 0x563cd29f51b0 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563cd2dd3fe0 con 0x563cd2a1e070 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.643+0000 7f825634d740 10 client.0 _ll_register_callbacks cb 0x563cd2cdf990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60628/0 (socket says 172.21.15.28:60628) 2024-09-21T05:52:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 -- 172.21.15.28:0/1281146906 learned_addr learned my addr 172.21.15.28:0/1281146906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2ddd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824db75640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dd1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 -- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 msgr2=0x563cd2ddd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2ddd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 -- 172.21.15.28:0/1281146906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 msgr2=0x563cd2dd1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dd1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824d374640 1 -- 172.21.15.28:0/1281146906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cd2cf3c40 con 0x563cd29f51b0 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.644+0000 7f824db75640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dd1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.645+0000 7f824d374640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8238014f40 tx=0x7f823800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5fe11e0feabf8f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.645+0000 7f82317fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.645+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823800dea0 con 0x563cd29f51b0 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.645+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8238014110 con 0x563cd29f51b0 2024-09-21T05:52:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.645+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8244004030 con 0x563cd29f51b0 2024-09-21T05:52:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.646+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8238004030 con 0x563cd29f51b0 2024-09-21T05:52:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.646+0000 7f825634d740 10 client.26402 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.646+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563cd2ddfdb0 con 0x563cd29f51b0 2024-09-21T05:52:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8244004030 con 0x563cd29f51b0 2024-09-21T05:52:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f8238029020 con 0x563cd29f51b0 2024-09-21T05:52:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 10 client.26402 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 10 client.26402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 20 client.26402 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd29dc9f0 0x563cd2de3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 4 client.26402 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.648+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cd2d3dda0 con 0x563cd29dc9f0 2024-09-21T05:52:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.650+0000 7f824db75640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd29dc9f0 0x563cd2de3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.651+0000 7f824db75640 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd29dc9f0 0x563cd2de3830 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4ab4895bfab3868 server_cookie=cb1365af20a75261 in_seq=0 out_seq=0 2024-09-21T05:52:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.652+0000 7f82317fa640 10 client.26402 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.653+0000 7f82317fa640 1 -- 172.21.15.28:0/1281146906 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563cd2d3dda0 con 0x563cd29dc9f0 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.653+0000 7f82317fa640 10 client.26402 handle_command_reply: tid=0 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.653+0000 7f825634d740 1 client.26402 shutdown 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 msgr2=0x563cd2dd0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8238014f40 tx=0x7f823800c850 comp rx=0 tx=0).stop 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 shutdown_connections 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd29dc9f0 0x563cd2de3830 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2ddd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dd1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 --2- 172.21.15.28:0/1281146906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2dd0ed0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 >> 172.21.15.28:0/1281146906 conn(0x563cd2a12460 msgr2=0x563cd2dc89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f82317fa640 0 client.26402 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 shutdown_connections 2024-09-21T05:52:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 1 -- 172.21.15.28:0/1281146906 wait complete. 2024-09-21T05:52:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.654+0000 7f825634d740 20 client.26402 trim_cache size 0 max 16384 2024-09-21T05:52:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.690+0000 7f825634d740 1 Processor -- start 2024-09-21T05:52:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.690+0000 7f825634d740 1 -- start start 2024-09-21T05:52:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.691+0000 7f825634d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2de37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.691+0000 7f825634d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.691+0000 7f825634d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2dc8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2de37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f824d374640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2de37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:60638/0 (socket says 172.21.15.28:60638) 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f824db75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f824db75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60304/0 (socket says 172.21.15.28:60304) 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f824d374640 1 -- 172.21.15.28:0/3050466124 learned_addr learned my addr 172.21.15.28:0/3050466124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00e7e0 con 0x563cd29fd390 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f822c010350 con 0x563cd29f51b0 2024-09-21T05:52:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.692+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00eeb0 con 0x563cd2a1e070 2024-09-21T05:52:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.693+0000 7f824db75640 1 -- 172.21.15.28:0/3050466124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 msgr2=0x563cd2dc8e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:52:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.693+0000 7f824db75640 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2dc8e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.693+0000 7f824db75640 1 -- 172.21.15.28:0/3050466124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 msgr2=0x563cd2de37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.693+0000 7f824db75640 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2de37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.693+0000 7f824db75640 1 -- 172.21.15.28:0/3050466124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cd2dcc760 con 0x563cd29fd390 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.694+0000 7f824db75640 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8244006010 tx=0x7f82440050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b586dd5a7c6acc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.695+0000 7f82337fe640 1 -- 172.21.15.28:0/3050466124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244046040 con 0x563cd29fd390 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.695+0000 7f82337fe640 1 -- 172.21.15.28:0/3050466124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8244004e10 con 0x563cd29fd390 2024-09-21T05:52:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.695+0000 7f82337fe640 1 -- 172.21.15.28:0/3050466124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82440035f0 con 0x563cd29fd390 2024-09-21T05:52:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 msgr2=0x563cd2dc8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8244006010 tx=0x7f82440050c0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 shutdown_connections 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd2a1e070 0x563cd2dc8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29fd390 0x563cd2dc8910 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 --2- 172.21.15.28:0/3050466124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd29f51b0 0x563cd2de37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.696+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 >> 172.21.15.28:0/3050466124 conn(0x563cd2a12460 msgr2=0x563cd2dcf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.697+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 shutdown_connections 2024-09-21T05:52:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.697+0000 7f825634d740 1 -- 172.21.15.28:0/3050466124 wait complete. 2024-09-21T05:52:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.697+0000 7f825634d740 1 -- start start 2024-09-21T05:52:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.698+0000 7f825634d740 1 Processor -- start 2024-09-21T05:52:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dd99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 0x563cd2dd4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2dd53c0 0x563cd2dd57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f824d374640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dd99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f824d374640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dd99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:58736/0 (socket says 172.21.15.28:58736) 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f824d374640 1 -- 172.21.15.28:0/3108767973 learned_addr learned my addr 172.21.15.28:0/3108767973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00fc80 con 0x563cd29b1da0 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00f580 con 0x563cd2dd53c0 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.699+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f822c00c7c0 con 0x563cd29fd390 2024-09-21T05:52:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f825634d740 10 client.0 _ll_register_callbacks cb 0x7f823800cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f824db75640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 0x563cd2dd4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f824d374640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2dd53c0 0x563cd2dd57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f824db75640 1 -- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 msgr2=0x563cd2dd99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f824db75640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dd99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.700+0000 7f824db75640 1 -- 172.21.15.28:0/3108767973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2dd53c0 msgr2=0x563cd2dd57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.701+0000 7f824db75640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2dd53c0 0x563cd2dd57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.701+0000 7f824db75640 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cd2e4d2e0 con 0x563cd29b1da0 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.701+0000 7f824db75640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 0x563cd2dd4e80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f824401a2b0 tx=0x7f8244004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9e856b5a1b6137 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.701+0000 7f82317fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.701+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244046040 con 0x563cd29b1da0 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.702+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f8244005130 con 0x563cd29b1da0 2024-09-21T05:52:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.702+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8244004e10 con 0x563cd29b1da0 2024-09-21T05:52:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.703+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8244075050 con 0x563cd29b1da0 2024-09-21T05:52:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.707+0000 7f825634d740 10 client.16824 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.707+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563cd2e4e530 con 0x563cd29b1da0 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.707+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8244004e10 con 0x563cd29b1da0 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f8244003b20 con 0x563cd29b1da0 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 10 client.16824 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 10 client.16824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 20 client.16824 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd2e4e810 0x563cd2e50c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 4 client.16824 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.708+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563cd2d39850 con 0x563cd2e4e810 2024-09-21T05:52:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.709+0000 7f824d374640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd2e4e810 0x563cd2e50c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.711+0000 7f824d374640 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd2e4e810 0x563cd2e50c30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e052d79d41704 server_cookie=f8e419ca2b25e579 in_seq=0 out_seq=0 2024-09-21T05:52:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.711+0000 7f82317fa640 10 client.16824 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.711+0000 7f82317fa640 1 -- 172.21.15.28:0/3108767973 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2362 (crc 0 0 0) 0x563cd2d39850 con 0x563cd2e4e810 2024-09-21T05:52:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.711+0000 7f82317fa640 10 client.16824 handle_command_reply: tid=0 2024-09-21T05:52:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.712+0000 7f825634d740 1 client.16824 shutdown 2024-09-21T05:52:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.712+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 msgr2=0x563cd2dd4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.712+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 0x563cd2dd4e80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f824401a2b0 tx=0x7f8244004130 comp rx=0 tx=0).stop 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.713+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 shutdown_connections 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x563cd2e4e810 0x563cd2e50c30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f82317fa640 0 client.16824 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563cd2dd53c0 0x563cd2dd57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cd29b1da0 0x563cd2dd4e80 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 --2- 172.21.15.28:0/3108767973 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563cd29fd390 0x563cd2dd99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 >> 172.21.15.28:0/3108767973 conn(0x563cd2a12460 msgr2=0x563cd2dcf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 shutdown_connections 2024-09-21T05:52:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.714+0000 7f825634d740 1 -- 172.21.15.28:0/3108767973 wait complete. 2024-09-21T05:52:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.717+0000 7f825634d740 20 client.16824 trim_cache size 0 max 16384 2024-09-21T05:52:24.719 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:52:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:52:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:52:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:52:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:52:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:52:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:52:24.725 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:52:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.717+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 msgr2=0x7f82340b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.717+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 0x7f82340b8d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8240016000 tx=0x7f8240013000 comp rx=0 tx=0).stop 2024-09-21T05:52:24.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.717+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 msgr2=0x7f8248113360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.717+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f824000efd0 tx=0x7f824000c6a0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 shutdown_connections 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82340b6850 0x7f82340b8d60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82481079f0 0x7f8248113360 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8248102f00 0x7f8248112dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 --2- 172.21.15.28:0/2462508591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82481025b0 0x7f8248116ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 >> 172.21.15.28:0/2462508591 conn(0x7f82480fe2c0 msgr2=0x7f8248105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 shutdown_connections 2024-09-21T05:52:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:24.719+0000 7f824eb77640 1 -- 172.21.15.28:0/2462508591 wait complete. 2024-09-21T05:52:26.044 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:52:26.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:52:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:26.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:26 smithi181 ceph-mon[34819]: pgmap v487: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 7.69k op/s 2024-09-21T05:52:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:26 smithi028 ceph-mon[31227]: pgmap v487: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 7.69k op/s 2024-09-21T05:52:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:26 smithi129 ceph-mon[34885]: pgmap v487: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 7.69k op/s 2024-09-21T05:52:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.107+0000 7f23bde9c640 1 -- 172.21.15.28:0/530567618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810fad0 msgr2=0x7f23b810fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.107+0000 7f23bde9c640 1 --2- 172.21.15.28:0/530567618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810fad0 0x7f23b810fee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009900 tx=0x7f23ac02a420 comp rx=0 tx=0).stop 2024-09-21T05:52:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 -- 172.21.15.28:0/530567618 shutdown_connections 2024-09-21T05:52:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 --2- 172.21.15.28:0/530567618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b8116c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 --2- 172.21.15.28:0/530567618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810fad0 0x7f23b810fee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 --2- 172.21.15.28:0/530567618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810a190 0x7f23b810a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 -- 172.21.15.28:0/530567618 >> 172.21.15.28:0/530567618 conn(0x7f23b80780f0 msgr2=0x7f23b81081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 -- 172.21.15.28:0/530567618 shutdown_connections 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.108+0000 7f23bde9c640 1 -- 172.21.15.28:0/530567618 wait complete. 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.109+0000 7f23bde9c640 1 Processor -- start 2024-09-21T05:52:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.109+0000 7f23bde9c640 1 -- start start 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 0x7f23b81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 0x7f23b81a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b81a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23b811a170 con 0x7f23b810fad0 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23b8119ff0 con 0x7f23b810a190 2024-09-21T05:52:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.110+0000 7f23bde9c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f23b811a2f0 con 0x7f23b8110420 2024-09-21T05:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.111+0000 7f23b7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b81a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.111+0000 7f23b7fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b81a6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57058/0 (socket says 172.21.15.28:57058) 2024-09-21T05:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.111+0000 7f23b7fff640 1 -- 172.21.15.28:0/262577389 learned_addr learned my addr 172.21.15.28:0/262577389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.111+0000 7f23b6ffd640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 0x7f23b81a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.111+0000 7f23b77fe640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 0x7f23b81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 -- 172.21.15.28:0/262577389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 msgr2=0x7f23b81a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b81a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 -- 172.21.15.28:0/262577389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 msgr2=0x7f23b81a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 0x7f23b81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 -- 172.21.15.28:0/262577389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b81a67f0 con 0x7f23b810fad0 2024-09-21T05:52:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.112+0000 7f23b6ffd640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 0x7f23b81a0ee0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02aa10 tx=0x7f23ac030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.113+0000 7f23b77fe640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 0x7f23b81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:52:27.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.114+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac03f070 con 0x7f23b810fad0 2024-09-21T05:52:27.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.114+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23ac004460 con 0x7f23b810fad0 2024-09-21T05:52:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.114+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac035030 con 0x7f23b810fad0 2024-09-21T05:52:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.116+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b81a6a80 con 0x7f23b810fad0 2024-09-21T05:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.116+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b81a6f60 con 0x7f23b810fad0 2024-09-21T05:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.117+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b81a1a70 con 0x7f23b810fad0 2024-09-21T05:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.121+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f23ac042070 con 0x7f23b810fad0 2024-09-21T05:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.122+0000 7f23b4ff9640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 0x7f23a00b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.122+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f23ac108020 con 0x7f23b810fad0 2024-09-21T05:52:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.123+0000 7f23b77fe640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 0x7f23a00b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.123+0000 7f23b77fe640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 0x7f23a00b8d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f23a8008b50 tx=0x7f23a8005e70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.129+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f23ac0ce5b0 con 0x7f23b810fad0 2024-09-21T05:52:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.566+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23b8110cc0 con 0x7f23b810fad0 2024-09-21T05:52:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.570+0000 7f23b4ff9640 1 -- 172.21.15.28:0/262577389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11988 (secure 0 0 0) 0x7f23ac0d2990 con 0x7f23b810fad0 2024-09-21T05:52:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:52:27.573 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"btime":"2024-09-21T05:51:47:129739+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:51:46.199369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:52:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 msgr2=0x7f23a00b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 0x7f23a00b8d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f23a8008b50 tx=0x7f23a8005e70 comp rx=0 tx=0).stop 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 msgr2=0x7f23b81a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 0x7f23b81a0ee0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02aa10 tx=0x7f23ac030650 comp rx=0 tx=0).stop 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 shutdown_connections 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23b8110420 0x7f23b81a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f23a00b6880 0x7f23a00b8d90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b810fad0 0x7f23b81a0ee0 secure :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02aa10 tx=0x7f23ac030650 comp rx=0 tx=0).stop 2024-09-21T05:52:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 --2- 172.21.15.28:0/262577389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b810a190 0x7f23b81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.576+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 >> 172.21.15.28:0/262577389 conn(0x7f23b80780f0 msgr2=0x7f23b810d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.577+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 shutdown_connections 2024-09-21T05:52:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:27.577+0000 7f23bde9c640 1 -- 172.21.15.28:0/262577389 wait complete. 2024-09-21T05:52:28.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:28 smithi181 ceph-mon[34819]: pgmap v488: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 33 KiB/s rd, 32 MiB/s wr, 7.55k op/s 2024-09-21T05:52:28.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:28 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/262577389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:52:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:28 smithi028 ceph-mon[31227]: pgmap v488: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 33 KiB/s rd, 32 MiB/s wr, 7.55k op/s 2024-09-21T05:52:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:28 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/262577389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:52:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:28 smithi129 ceph-mon[34885]: pgmap v488: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 33 KiB/s rd, 32 MiB/s wr, 7.55k op/s 2024-09-21T05:52:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:28 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/262577389' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:52:29.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:52:29.779 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 -- 172.21.15.28:0/4124082587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa65810b6d0 msgr2=0x7fa65810bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 --2- 172.21.15.28:0/4124082587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa65810b6d0 0x7fa65810bae0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009900 tx=0x7fa64c02a400 comp rx=0 tx=0).stop 2024-09-21T05:52:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 -- 172.21.15.28:0/4124082587 shutdown_connections 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 --2- 172.21.15.28:0/4124082587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 --2- 172.21.15.28:0/4124082587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa65810b6d0 0x7fa65810bae0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 --2- 172.21.15.28:0/4124082587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa658105d90 0x7fa6581061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 -- 172.21.15.28:0/4124082587 >> 172.21.15.28:0/4124082587 conn(0x7fa6581017e0 msgr2=0x7fa658103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 -- 172.21.15.28:0/4124082587 shutdown_connections 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.376+0000 7fa65d263640 1 -- 172.21.15.28:0/4124082587 wait complete. 2024-09-21T05:52:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.377+0000 7fa65d263640 1 Processor -- start 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.377+0000 7fa65d263640 1 -- start start 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa658105d90 0x7fa6581127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 0x7fa65810fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa658115d50 con 0x7fa65810c020 2024-09-21T05:52:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa658115bd0 con 0x7fa65810b6d0 2024-09-21T05:52:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa65d263640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa658115ed0 con 0x7fa658105d90 2024-09-21T05:52:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46862/0 (socket says 172.21.15.28:46862) 2024-09-21T05:52:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa6577fe640 1 -- 172.21.15.28:0/1282912336 learned_addr learned my addr 172.21.15.28:0/1282912336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.378+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 0x7fa65810fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 -- 172.21.15.28:0/1282912336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa658105d90 msgr2=0x7fa6581127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa658105d90 0x7fa6581127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 -- 172.21.15.28:0/1282912336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 msgr2=0x7fa65810fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 0x7fa65810fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 -- 172.21.15.28:0/1282912336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6581aca90 con 0x7fa65810c020 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 0x7fa65810fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.379+0000 7fa6577fe640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b970 tx=0x7fa65000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.380+0000 7fa63ffff640 1 -- 172.21.15.28:0/1282912336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650019070 con 0x7fa65810c020 2024-09-21T05:52:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.380+0000 7fa63ffff640 1 -- 172.21.15.28:0/1282912336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa650014440 con 0x7fa65810c020 2024-09-21T05:52:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.382+0000 7fa63ffff640 1 -- 172.21.15.28:0/1282912336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500075d0 con 0x7fa65810c020 2024-09-21T05:52:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.382+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6581acd20 con 0x7fa65810c020 2024-09-21T05:52:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.382+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581ad1e0 con 0x7fa65810c020 2024-09-21T05:52:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.386+0000 7fa6637d7740 1 Processor -- start 2024-09-21T05:52:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.388+0000 7fa63ffff640 1 -- 172.21.15.28:0/1282912336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa65000cb40 con 0x7fa65810c020 2024-09-21T05:52:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.389+0000 7fa63ffff640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 0x7fa6400b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.389+0000 7fa63ffff640 1 -- 172.21.15.28:0/1282912336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa6500e6ae0 con 0x7fa65810c020 2024-09-21T05:52:30.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.389+0000 7fa6637d7740 1 -- start start 2024-09-21T05:52:30.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 0x7fa6400b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390ee070 0x5594394952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x559439495800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55943940bea0 con 0x5594390cd390 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5594393fb2d0 con 0x5594390c51b0 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6637d7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5594393ff0e0 con 0x5594390ee070 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46870/0 (socket says 172.21.15.28:46870) 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 -- 172.21.15.28:0/3824679134 learned_addr learned my addr 172.21.15.28:0/3824679134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa656ffd640 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390ee070 0x5594394952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 -- 172.21.15.28:0/3824679134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390ee070 msgr2=0x5594394952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390ee070 0x5594394952c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 -- 172.21.15.28:0/3824679134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 msgr2=0x559439495800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x559439495800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.390+0000 7fa6577fe640 1 -- 172.21.15.28:0/3824679134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559439159650 con 0x5594390cd390 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.391+0000 7fa6577fe640 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ac8e0 tx=0x7fa6500ac360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd509526f5a3723 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.391+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 0x7fa6400b8af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa64c02a920 tx=0x7fa64c00b040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.392+0000 7fa63d7fa640 1 -- 172.21.15.28:0/3824679134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500ad050 con 0x5594390cd390 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.392+0000 7fa63d7fa640 1 -- 172.21.15.28:0/3824679134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6500b1430 con 0x5594390cd390 2024-09-21T05:52:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.392+0000 7fa63d7fa640 1 -- 172.21.15.28:0/3824679134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500a8500 con 0x5594390cd390 2024-09-21T05:52:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 -- 172.21.15.28:0/3824679134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 msgr2=0x559439495d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ac8e0 tx=0x7fa6500ac360 comp rx=0 tx=0).stop 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 -- 172.21.15.28:0/3824679134 shutdown_connections 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390cd390 0x559439495d40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x559439495800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 --2- 172.21.15.28:0/3824679134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390ee070 0x5594394952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 -- 172.21.15.28:0/3824679134 >> 172.21.15.28:0/3824679134 conn(0x5594390e2460 msgr2=0x5594394931b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.393+0000 7fa6637d7740 1 -- 172.21.15.28:0/3824679134 shutdown_connections 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.394+0000 7fa6637d7740 1 -- 172.21.15.28:0/3824679134 wait complete. 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.394+0000 7fa6637d7740 1 -- start start 2024-09-21T05:52:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.394+0000 7fa6637d7740 1 Processor -- start 2024-09-21T05:52:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x5594394a0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 0x5594394a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 0x5594394acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55943925c700 con 0x5594390ee070 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x5594394a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559439396160 con 0x5594390c51b0 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x5594394a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:56056/0 (socket says 172.21.15.28:56056) 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa656ffd640 1 -- 172.21.15.28:0/631033972 learned_addr learned my addr 172.21.15.28:0/631033972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5594394a3a40 con 0x5594390cd390 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.395+0000 7fa6567fc640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 0x5594394a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6637d7740 10 client.0 _ll_register_callbacks cb 0x5594393af9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 0x5594394acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 -- 172.21.15.28:0/631033972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 msgr2=0x5594394a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 0x5594394a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 -- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 msgr2=0x5594394a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x5594394a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 -- 172.21.15.28:0/631033972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594393c3c90 con 0x5594390ee070 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6567fc640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 0x5594394a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:52:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa6577fe640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 0x5594394acd30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ac8e0 tx=0x7fa6500a8e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e93310348e03ad server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.396+0000 7fa63aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.397+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500ad050 con 0x5594390ee070 2024-09-21T05:52:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.397+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa6500abdc0 con 0x5594390ee070 2024-09-21T05:52:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.397+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6500b1430 con 0x5594390ee070 2024-09-21T05:52:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa6500c2b00 con 0x5594390ee070 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa6637d7740 10 client.16836 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5594394af760 con 0x5594390ee070 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6500b1430 con 0x5594390ee070 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fa6500b25c0 con 0x5594390ee070 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa6637d7740 10 client.16836 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.398+0000 7fa6637d7740 10 client.16836 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.399+0000 7fa6637d7740 20 client.16836 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.399+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ac9f0 0x5594394b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.399+0000 7fa6637d7740 4 client.16836 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.399+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55943940ddf0 con 0x5594390ac9f0 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.399+0000 7fa656ffd640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ac9f0 0x5594394b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.400+0000 7fa63aff5640 10 client.16836 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:52:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.400+0000 7fa656ffd640 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ac9f0 0x5594394b2ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6557c195b36f25e9 server_cookie=41cf84c736ef80a0 in_seq=0 out_seq=0 2024-09-21T05:52:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.401+0000 7fa63aff5640 1 -- 172.21.15.28:0/631033972 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55943940ddf0 con 0x5594390ac9f0 2024-09-21T05:52:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.401+0000 7fa63aff5640 10 client.16836 handle_command_reply: tid=0 2024-09-21T05:52:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.403+0000 7fa6637d7740 1 client.16836 shutdown 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.403+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 msgr2=0x5594394acd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.403+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 0x5594394acd30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa6500ac8e0 tx=0x7fa6500a8e50 comp rx=0 tx=0).stop 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 shutdown_connections 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ac9f0 0x5594394b2ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390ee070 0x5594394acd30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa63aff5640 0 client.16836 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390cd390 0x5594394a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 --2- 172.21.15.28:0/631033972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390c51b0 0x5594394a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.404+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 >> 172.21.15.28:0/631033972 conn(0x5594390e2460 msgr2=0x559439498450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.406+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 shutdown_connections 2024-09-21T05:52:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.406+0000 7fa6637d7740 1 -- 172.21.15.28:0/631033972 wait complete. 2024-09-21T05:52:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.406+0000 7fa6637d7740 20 client.16836 trim_cache size 0 max 16384 2024-09-21T05:52:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 Processor -- start 2024-09-21T05:52:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 -- start start 2024-09-21T05:52:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x5594394962f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b5ae0 con 0x5594390c51b0 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b7650 con 0x5594390cd390 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.429+0000 7fa6637d7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b61b0 con 0x5594390e2460 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:56066/0 (socket says 172.21.15.28:56066) 2024-09-21T05:52:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 -- 172.21.15.28:0/4189644563 learned_addr learned my addr 172.21.15.28:0/4189644563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 -- 172.21.15.28:0/4189644563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 msgr2=0x5594394962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x5594394962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 -- 172.21.15.28:0/4189644563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 msgr2=0x5594394a2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 -- 172.21.15.28:0/4189644563 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559439493aa0 con 0x5594390cd390 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.430+0000 7fa6577fe640 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa6500e4300 tx=0x7fa6500ac360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f02f091583b59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.431+0000 7fa63cff9640 1 -- 172.21.15.28:0/4189644563 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500a8fa0 con 0x5594390cd390 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.431+0000 7fa63cff9640 1 -- 172.21.15.28:0/4189644563 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6500ad050 con 0x5594390cd390 2024-09-21T05:52:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.431+0000 7fa63cff9640 1 -- 172.21.15.28:0/4189644563 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6500abdc0 con 0x5594390cd390 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.432+0000 7fa6637d7740 1 -- 172.21.15.28:0/4189644563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 msgr2=0x5594394acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.432+0000 7fa6637d7740 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa6500e4300 tx=0x7fa6500ac360 comp rx=0 tx=0).stop 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.432+0000 7fa6637d7740 1 -- 172.21.15.28:0/4189644563 shutdown_connections 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.432+0000 7fa6637d7740 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394acca0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.432+0000 7fa6637d7740 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394a2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.433+0000 7fa6637d7740 1 --2- 172.21.15.28:0/4189644563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x5594394962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.433+0000 7fa6637d7740 1 -- 172.21.15.28:0/4189644563 >> 172.21.15.28:0/4189644563 conn(0x5594394a1050 msgr2=0x55943949a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.433+0000 7fa6637d7740 1 -- 172.21.15.28:0/4189644563 shutdown_connections 2024-09-21T05:52:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.433+0000 7fa6637d7740 1 -- 172.21.15.28:0/4189644563 wait complete. 2024-09-21T05:52:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.433+0000 7fa6637d7740 1 -- start start 2024-09-21T05:52:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 Processor -- start 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x55943951c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b6f80 con 0x5594390c51b0 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b6880 con 0x5594390cd390 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581b3ac0 con 0x5594390e2460 2024-09-21T05:52:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.434+0000 7fa6637d7740 10 client.0 _ll_register_callbacks cb 0x7fa64800b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46904/0 (socket says 172.21.15.28:46904) 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 -- 172.21.15.28:0/1281243819 learned_addr learned my addr 172.21.15.28:0/1281243819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa6577fe640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x55943951c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 -- 172.21.15.28:0/1281243819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 msgr2=0x55943951c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x55943951c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 -- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 msgr2=0x5594394b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 -- 172.21.15.28:0/1281243819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594394b4b40 con 0x5594390c51b0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa656ffd640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa648005d00 tx=0x7fa64800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672b4e6afba71e00 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.435+0000 7fa63aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648003df0 con 0x5594390c51b0 2024-09-21T05:52:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6500ad050 con 0x5594390c51b0 2024-09-21T05:52:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa648005020 con 0x5594390c51b0 2024-09-21T05:52:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa648005220 con 0x5594390c51b0 2024-09-21T05:52:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa6637d7740 10 client.16839 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55943951d870 con 0x5594390c51b0 2024-09-21T05:52:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.436+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6500ad050 con 0x5594390c51b0 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fa64802a070 con 0x5594390c51b0 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 10 client.16839 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 10 client.16839 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 20 client.16839 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ee070 0x559439520060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 4 client.16839 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594394098a0 con 0x5594390ee070 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.437+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ee070 0x559439520060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.438+0000 7fa63aff5640 10 client.16839 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.438+0000 7fa6567fc640 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ee070 0x559439520060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae1e2917c3c1a715 server_cookie=7387ad82d303ef9e in_seq=0 out_seq=0 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.438+0000 7fa63aff5640 1 -- 172.21.15.28:0/1281243819 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594394098a0 con 0x5594390ee070 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.438+0000 7fa63aff5640 10 client.16839 handle_command_reply: tid=0 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.438+0000 7fa6637d7740 1 client.16839 shutdown 2024-09-21T05:52:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 msgr2=0x5594394b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa648005d00 tx=0x7fa64800be20 comp rx=0 tx=0).stop 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 shutdown_connections 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5594390ee070 0x559439520060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594390e2460 0x55943951c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594390cd390 0x5594394b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 --2- 172.21.15.28:0/1281243819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594390c51b0 0x5594394b3db0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 >> 172.21.15.28:0/1281243819 conn(0x5594394a1050 msgr2=0x5594394a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa63aff5640 0 client.16839 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 shutdown_connections 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 1 -- 172.21.15.28:0/1281243819 wait complete. 2024-09-21T05:52:30.444 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:52:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.439+0000 7fa6637d7740 20 client.16839 trim_cache size 0 max 16384 2024-09-21T05:52:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 msgr2=0x7fa6400b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 0x7fa6400b8af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa64c02a920 tx=0x7fa64c00b040 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 msgr2=0x7fa658110220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b970 tx=0x7fa65000be30 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 shutdown_connections 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6400b65e0 0x7fa6400b8af0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.440+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa65810c020 0x7fa658110220 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.441+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa65810b6d0 0x7fa65810fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.441+0000 7fa65d263640 1 --2- 172.21.15.28:0/1282912336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa658105d90 0x7fa6581127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.441+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 >> 172.21.15.28:0/1282912336 conn(0x7fa6581017e0 msgr2=0x7fa658108b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.441+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 shutdown_connections 2024-09-21T05:52:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:30.441+0000 7fa65d263640 1 -- 172.21.15.28:0/1282912336 wait complete. 2024-09-21T05:52:30.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:30 smithi181 ceph-mon[34819]: pgmap v489: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1017 GiB / 1.0 TiB avail; 46 KiB/s rd, 34 MiB/s wr, 7.96k op/s 2024-09-21T05:52:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:30 smithi028 ceph-mon[31227]: pgmap v489: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1017 GiB / 1.0 TiB avail; 46 KiB/s rd, 34 MiB/s wr, 7.96k op/s 2024-09-21T05:52:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:30 smithi129 ceph-mon[34885]: pgmap v489: 97 pgs: 97 active+clean; 16 GiB data, 55 GiB used, 1017 GiB / 1.0 TiB avail; 46 KiB/s rd, 34 MiB/s wr, 7.96k op/s 2024-09-21T05:52:31.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:52:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.506+0000 7fca4636d640 1 -- 172.21.15.28:0/1108729269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca401021f0 msgr2=0x7fca40102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.506+0000 7fca4636d640 1 --2- 172.21.15.28:0/1108729269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca401021f0 0x7fca40102600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fca30009900 tx=0x7fca3002b510 comp rx=0 tx=0).stop 2024-09-21T05:52:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.507+0000 7fca4636d640 1 -- 172.21.15.28:0/1108729269 shutdown_connections 2024-09-21T05:52:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.507+0000 7fca4636d640 1 --2- 172.21.15.28:0/1108729269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 0x7fca4010ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.507+0000 7fca4636d640 1 --2- 172.21.15.28:0/1108729269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca40107b30 0x7fca40107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.507+0000 7fca4636d640 1 --2- 172.21.15.28:0/1108729269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca401021f0 0x7fca40102600 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.507+0000 7fca4636d640 1 -- 172.21.15.28:0/1108729269 >> 172.21.15.28:0/1108729269 conn(0x7fca40077c60 msgr2=0x7fca400780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.508+0000 7fca4636d640 1 -- 172.21.15.28:0/1108729269 shutdown_connections 2024-09-21T05:52:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.508+0000 7fca4636d640 1 -- 172.21.15.28:0/1108729269 wait complete. 2024-09-21T05:52:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 Processor -- start 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 -- start start 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca401021f0 0x7fca40074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 0x7fca4006f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca4006d6a0 con 0x7fca40108480 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca4006d520 con 0x7fca40107b30 2024-09-21T05:52:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.510+0000 7fca4636d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca4006d820 con 0x7fca401021f0 2024-09-21T05:52:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.512+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.512+0000 7fca4490a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 0x7fca4006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.512+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:56086/0 (socket says 172.21.15.28:56086) 2024-09-21T05:52:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.512+0000 7fca3f7fe640 1 -- 172.21.15.28:0/2091123898 learned_addr learned my addr 172.21.15.28:0/2091123898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.514+0000 7fca3f7fe640 1 -- 172.21.15.28:0/2091123898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca401021f0 msgr2=0x7fca40074ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:52:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.514+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca401021f0 0x7fca40074ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.514+0000 7fca3f7fe640 1 -- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 msgr2=0x7fca4006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.514+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 0x7fca4006f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.514+0000 7fca3f7fe640 1 -- 172.21.15.28:0/2091123898 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4006ffb0 con 0x7fca40107b30 2024-09-21T05:52:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.517+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fca3400c970 tx=0x7fca3400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.523+0000 7fca3d7fa640 1 -- 172.21.15.28:0/2091123898 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34007bb0 con 0x7fca40107b30 2024-09-21T05:52:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.523+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca401acec0 con 0x7fca40107b30 2024-09-21T05:52:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.523+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca401ad3d0 con 0x7fca40107b30 2024-09-21T05:52:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.523+0000 7fca3d7fa640 1 -- 172.21.15.28:0/2091123898 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca34007d50 con 0x7fca40107b30 2024-09-21T05:52:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.523+0000 7fca3d7fa640 1 -- 172.21.15.28:0/2091123898 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3400f450 con 0x7fca40107b30 2024-09-21T05:52:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.530+0000 7fca3d7fa640 1 -- 172.21.15.28:0/2091123898 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fca3400f5f0 con 0x7fca40107b30 2024-09-21T05:52:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.531+0000 7fca3d7fa640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 0x7fca280b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.531+0000 7fca4c8e1740 1 Processor -- start 2024-09-21T05:52:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 -- start start 2024-09-21T05:52:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043597070 0x55e04393e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04393ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 0x55e04393f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0438b4ea0 con 0x55e043597070 2024-09-21T05:52:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:32 smithi028 ceph-mon[31227]: pgmap v490: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 5.17k op/s 2024-09-21T05:52:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e0438a42d0 con 0x55e04356e1b0 2024-09-21T05:52:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e0438a80e0 con 0x55e043576390 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04393ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04393ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:56094/0 (socket says 172.21.15.28:56094) 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca3f7fe640 1 -- 172.21.15.28:0/652850263 learned_addr learned my addr 172.21.15.28:0/652850263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.532+0000 7fca4490a640 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 0x55e04393f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.533+0000 7fca4490a640 1 -- 172.21.15.28:0/652850263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 msgr2=0x55e04393ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.533+0000 7fca4490a640 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04393ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.533+0000 7fca4490a640 1 -- 172.21.15.28:0/652850263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043597070 msgr2=0x55e04393e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.533+0000 7fca4490a640 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043597070 0x55e04393e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.533+0000 7fca4490a640 1 -- 172.21.15.28:0/652850263 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e043602650 con 0x55e043576390 2024-09-21T05:52:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.534+0000 7fca3d7fa640 1 -- 172.21.15.28:0/2091123898 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fca340e8300 con 0x7fca40107b30 2024-09-21T05:52:32.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:32 smithi181 ceph-mon[34819]: pgmap v490: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 5.17k op/s 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.536+0000 7fca4490a640 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 0x55e04393f120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca38009eb0 tx=0x7fca38010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e499a23ab27bd99 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.537+0000 7fca267fc640 1 -- 172.21.15.28:0/652850263 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38016070 con 0x55e043576390 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.537+0000 7fca267fc640 1 -- 172.21.15.28:0/652850263 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca38004d20 con 0x55e043576390 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.538+0000 7fca4c8e1740 1 -- 172.21.15.28:0/652850263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 msgr2=0x55e04393f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.538+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 0x55e04393f120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca38009eb0 tx=0x7fca38010740 comp rx=0 tx=0).stop 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.539+0000 7fca4c8e1740 1 -- 172.21.15.28:0/652850263 shutdown_connections 2024-09-21T05:52:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.539+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043576390 0x55e04393f120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.539+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04393ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.539+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/652850263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043597070 0x55e04393e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.539+0000 7fca4c8e1740 1 -- 172.21.15.28:0/652850263 >> 172.21.15.28:0/652850263 conn(0x55e04358b460 msgr2=0x55e04393c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.544+0000 7fca3ffff640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 0x7fca280b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.551+0000 7fca3ffff640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 0x7fca280b8d90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fca30009850 tx=0x7fca30005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.552+0000 7fca4c8e1740 1 -- 172.21.15.28:0/652850263 shutdown_connections 2024-09-21T05:52:32.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.554+0000 7fca4c8e1740 1 -- 172.21.15.28:0/652850263 wait complete. 2024-09-21T05:52:32.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.555+0000 7fca4c8e1740 1 -- start start 2024-09-21T05:52:32.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.555+0000 7fca4c8e1740 1 Processor -- start 2024-09-21T05:52:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e04356e1b0 0x55e04394c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0435559f0 0x55e04394a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e043705700 con 0x55e04356e1b0 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e04383f160 con 0x55e0435559f0 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e04394ced0 con 0x55e043597070 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca4c8e1740 10 client.0 _ll_register_callbacks cb 0x55e0438589e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57154/0 (socket says 172.21.15.28:57154) 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.556+0000 7fca3f7fe640 1 -- 172.21.15.28:0/3261553678 learned_addr learned my addr 172.21.15.28:0/3261553678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3ffff640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e04356e1b0 0x55e04394c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3f7fe640 1 -- 172.21.15.28:0/3261553678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0435559f0 msgr2=0x55e04394a170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0435559f0 0x55e04394a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3f7fe640 1 -- 172.21.15.28:0/3261553678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e04356e1b0 msgr2=0x55e04394c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e04356e1b0 0x55e04394c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.557+0000 7fca3f7fe640 1 -- 172.21.15.28:0/3261553678 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e04386cc90 con 0x55e043597070 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.558+0000 7fca3f7fe640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fca340aa6e0 tx=0x7fca34000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8adb0528b6f5912e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.558+0000 7fca1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.558+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3401d070 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.559+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e0438b6df0 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.559+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fca340b78b0 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.559+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fca340ad880 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.559+0000 7fca4c8e1740 10 client.26446 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.559+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e0439589a0 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.560+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e0438b6df0 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.560+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fca34099140 con 0x55e043597070 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.560+0000 7fca4c8e1740 10 client.26446 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.560+0000 7fca4c8e1740 10 client.26446 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.560+0000 7fca4c8e1740 20 client.26446 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:32.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.561+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e04352ada0 0x55e04395c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.561+0000 7fca4c8e1740 4 client.26446 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:32.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.561+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0438b28a0 con 0x55e04352ada0 2024-09-21T05:52:32.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.562+0000 7fca3ffff640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e04352ada0 0x55e04395c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.570+0000 7fca1f7fe640 10 client.26446 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:52:32.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.570+0000 7fca3ffff640 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e04352ada0 0x55e04395c460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0453265fccdd78c server_cookie=2f448a48bc93c63d in_seq=0 out_seq=0 2024-09-21T05:52:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.574+0000 7fca1f7fe640 1 -- 172.21.15.28:0/3261553678 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e0438b28a0 con 0x55e04352ada0 2024-09-21T05:52:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.574+0000 7fca1f7fe640 10 client.26446 handle_command_reply: tid=0 2024-09-21T05:52:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.574+0000 7fca4c8e1740 1 client.26446 shutdown 2024-09-21T05:52:32.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 msgr2=0x55e043949a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fca340aa6e0 tx=0x7fca34000bc0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 shutdown_connections 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e04352ada0 0x55e04395c460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0435559f0 0x55e04394a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043597070 0x55e043949a60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3261553678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e04356e1b0 0x55e04394c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 >> 172.21.15.28:0/3261553678 conn(0x55e04358b460 msgr2=0x55e043941880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca1f7fe640 0 client.26446 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 shutdown_connections 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3261553678 wait complete. 2024-09-21T05:52:32.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.582+0000 7fca4c8e1740 20 client.26446 trim_cache size 0 max 16384 2024-09-21T05:52:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:32 smithi129 ceph-mon[34885]: pgmap v490: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 5.17k op/s 2024-09-21T05:52:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.615+0000 7fca4c8e1740 1 Processor -- start 2024-09-21T05:52:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.616+0000 7fca4c8e1740 1 -- start start 2024-09-21T05:52:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.616+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e0435559f0 0x55e0439417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.616+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e043941ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.617+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 0x55e043946bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.617+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00e7c0 con 0x55e043942220 2024-09-21T05:52:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.617+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c010330 con 0x55e04356e1b0 2024-09-21T05:52:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.617+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00ee90 con 0x55e0435559f0 2024-09-21T05:52:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.618+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e043941ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.618+0000 7fca3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e043941ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:56120/0 (socket says 172.21.15.28:56120) 2024-09-21T05:52:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.618+0000 7fca3f7fe640 1 -- 172.21.15.28:0/3027795430 learned_addr learned my addr 172.21.15.28:0/3027795430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.619+0000 7fca4490a640 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 0x55e043946bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.619+0000 7fca3ffff640 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e0435559f0 0x55e0439417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.621+0000 7fca4490a640 1 -- 172.21.15.28:0/3027795430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e0435559f0 msgr2=0x55e0439417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.621+0000 7fca4490a640 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e0435559f0 0x55e0439417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.621+0000 7fca4490a640 1 -- 172.21.15.28:0/3027795430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 msgr2=0x55e043941ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.621+0000 7fca4490a640 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e043941ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.621+0000 7fca4490a640 1 -- 172.21.15.28:0/3027795430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e043945550 con 0x55e043942220 2024-09-21T05:52:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.623+0000 7fca4490a640 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 0x55e043946bd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fca38006010 tx=0x7fca38010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed425d0f8ad42b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.629+0000 7fca25ffb640 1 -- 172.21.15.28:0/3027795430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38027070 con 0x55e043942220 2024-09-21T05:52:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.629+0000 7fca25ffb640 1 -- 172.21.15.28:0/3027795430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca38012850 con 0x55e043942220 2024-09-21T05:52:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.629+0000 7fca25ffb640 1 -- 172.21.15.28:0/3027795430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38016070 con 0x55e043942220 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.630+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3027795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 msgr2=0x55e043946bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.630+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 0x55e043946bd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fca38006010 tx=0x7fca38010f10 comp rx=0 tx=0).stop 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.631+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3027795430 shutdown_connections 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.631+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e043942220 0x55e043946bd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.631+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e043941ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.631+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/3027795430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e0435559f0 0x55e0439417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.631+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3027795430 >> 172.21.15.28:0/3027795430 conn(0x55e04358b460 msgr2=0x55e043948260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.632+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3027795430 shutdown_connections 2024-09-21T05:52:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.632+0000 7fca4c8e1740 1 -- 172.21.15.28:0/3027795430 wait complete. 2024-09-21T05:52:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.633+0000 7fca4c8e1740 1 -- start start 2024-09-21T05:52:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.633+0000 7fca4c8e1740 1 Processor -- start 2024-09-21T05:52:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.634+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0435559f0 0x55e04394e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.634+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04394ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4c8e1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00fc60 con 0x55e0435559f0 2024-09-21T05:52:32.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4c8e1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00f560 con 0x55e04356e1b0 2024-09-21T05:52:32.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4490a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4490a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:57166/0 (socket says 172.21.15.28:57166) 2024-09-21T05:52:32.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4490a640 1 -- 172.21.15.28:0/2344449446 learned_addr learned my addr 172.21.15.28:0/2344449446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:32.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.635+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00c7a0 con 0x55e043942220 2024-09-21T05:52:32.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.636+0000 7fca4c8e1740 10 client.0 _ll_register_callbacks cb 0x7fca340aa0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:32.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.636+0000 7fca3ffff640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0435559f0 0x55e04394e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.637+0000 7fca4490a640 1 -- 172.21.15.28:0/2344449446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 msgr2=0x55e04394ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.637+0000 7fca4490a640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04394ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.637+0000 7fca4490a640 1 -- 172.21.15.28:0/2344449446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0435559f0 msgr2=0x55e04394e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.637+0000 7fca4490a640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0435559f0 0x55e04394e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.637+0000 7fca4490a640 1 -- 172.21.15.28:0/2344449446 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0439c63a0 con 0x55e043942220 2024-09-21T05:52:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.645+0000 7fca4490a640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fca38012cd0 tx=0x7fca38012d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=119b0194689dc623 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.646+0000 7fca1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.646+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38032060 con 0x55e043942220 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.647+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fca38003b70 con 0x55e043942220 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.647+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca38012850 con 0x55e043942220 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.648+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fca38027040 con 0x55e043942220 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.651+0000 7fca4c8e1740 10 client.26449 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.651+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e0439c7490 con 0x55e043942220 2024-09-21T05:52:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.655+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca38003d70 con 0x55e043942220 2024-09-21T05:52:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.655+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fca38021030 con 0x55e043942220 2024-09-21T05:52:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 10 client.26449 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 10 client.26449 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 20 client.26449 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e0439c78e0 0x55e0439c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 4 client.26449 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fca38012850 con 0x55e0439c78e0 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.656+0000 7fca3ffff640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e0439c78e0 0x55e0439c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.657+0000 7fca1f7fe640 10 client.26449 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:52:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca3ffff640 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e0439c78e0 0x55e0439c9d00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=43f096a640a3ca45 server_cookie=953236e7d29efba5 in_seq=0 out_seq=0 2024-09-21T05:52:32.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca1f7fe640 1 -- 172.21.15.28:0/2344449446 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fca38012850 con 0x55e0439c78e0 2024-09-21T05:52:32.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca1f7fe640 10 client.26449 handle_command_reply: tid=0 2024-09-21T05:52:32.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca4c8e1740 1 client.26449 shutdown 2024-09-21T05:52:32.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 msgr2=0x55e0439c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.659+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fca38012cd0 tx=0x7fca38012d00 comp rx=0 tx=0).stop 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 shutdown_connections 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55e0439c78e0 0x55e0439c9d00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e043942220 0x55e0439c5e60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca1f7fe640 0 client.26449 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e04356e1b0 0x55e04394ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 --2- 172.21.15.28:0/2344449446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0435559f0 0x55e04394e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.662+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 >> 172.21.15.28:0/2344449446 conn(0x55e04358b460 msgr2=0x55e043947e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.663+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 shutdown_connections 2024-09-21T05:52:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.663+0000 7fca4c8e1740 1 -- 172.21.15.28:0/2344449446 wait complete. 2024-09-21T05:52:32.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.664+0000 7fca4c8e1740 20 client.26449 trim_cache size 0 max 16384 2024-09-21T05:52:32.668 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:52:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 msgr2=0x7fca280b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 0x7fca280b8d90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fca30009850 tx=0x7fca30005fd0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 msgr2=0x7fca4006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fca3400c970 tx=0x7fca3400ce30 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 shutdown_connections 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fca280b6880 0x7fca280b8d90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca40108480 0x7fca4006f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca40107b30 0x7fca4006f1a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 --2- 172.21.15.28:0/2091123898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca401021f0 0x7fca40074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.670+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 >> 172.21.15.28:0/2091123898 conn(0x7fca40077c60 msgr2=0x7fca40105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:32.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.674+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 shutdown_connections 2024-09-21T05:52:32.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:32.674+0000 7fca4636d640 1 -- 172.21.15.28:0/2091123898 wait complete. 2024-09-21T05:52:34.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:34 smithi028 ceph-mon[31227]: pgmap v491: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-09-21T05:52:34.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:34 smithi129 ceph-mon[34885]: pgmap v491: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-09-21T05:52:34.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:34 smithi181 ceph-mon[34819]: pgmap v491: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 4.38k op/s 2024-09-21T05:52:34.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:52:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:35.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 -- 172.21.15.28:0/656901178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10b620 msgr2=0x7f1cdc10ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 --2- 172.21.15.28:0/656901178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10b620 0x7f1cdc10ba30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009860 tx=0x7f1ccc02a340 comp rx=0 tx=0).stop 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 -- 172.21.15.28:0/656901178 shutdown_connections 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 --2- 172.21.15.28:0/656901178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1127a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 --2- 172.21.15.28:0/656901178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10b620 0x7f1cdc10ba30 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009860 tx=0x7f1ccc02a340 comp rx=0 tx=0).stop 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 --2- 172.21.15.28:0/656901178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc106100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.676+0000 7f1ce125c640 1 -- 172.21.15.28:0/656901178 >> 172.21.15.28:0/656901178 conn(0x7f1cdc1016f0 msgr2=0x7f1cdc103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.677+0000 7f1ce125c640 1 -- 172.21.15.28:0/656901178 shutdown_connections 2024-09-21T05:52:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.677+0000 7f1ce125c640 1 -- 172.21.15.28:0/656901178 wait complete. 2024-09-21T05:52:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.679+0000 7f1ce125c640 1 Processor -- start 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.679+0000 7f1ce125c640 1 -- start start 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.679+0000 7f1ce125c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc10fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.679+0000 7f1ce125c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1ce125c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 0x7f1cdc1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1ce125c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc115c30 con 0x7f1cdc1107e0 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1ce125c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc115ab0 con 0x7f1cdc10bf70 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1ce125c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc115db0 con 0x7f1cdc105cf0 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cda59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 0x7f1cdc1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cda59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38250/0 (socket says 172.21.15.28:38250) 2024-09-21T05:52:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cda59c640 1 -- 172.21.15.28:0/2117785905 learned_addr learned my addr 172.21.15.28:0/2117785905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 -- 172.21.15.28:0/2117785905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 msgr2=0x7f1cdc10fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc10fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc10fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 -- 172.21.15.28:0/2117785905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 msgr2=0x7f1cdc1102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdb59e640 1 -- 172.21.15.28:0/2117785905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cdc1a2870 con 0x7f1cdc1107e0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.680+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1cdb59e640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 0x7f1cdc1a20f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400efc0 tx=0x7f1cd400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1cda59c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1cc3fff640 1 -- 172.21.15.28:0/2117785905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4003970 con 0x7f1cdc1107e0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cdc1a2b60 con 0x7f1cdc1107e0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cdc1a30b0 con 0x7f1cdc1107e0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.681+0000 7f1cc3fff640 1 -- 172.21.15.28:0/2117785905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd4004020 con 0x7f1cdc1107e0 2024-09-21T05:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.683+0000 7f1cc3fff640 1 -- 172.21.15.28:0/2117785905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd400b900 con 0x7f1cdc1107e0 2024-09-21T05:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.685+0000 7f1ce81cb740 1 Processor -- start 2024-09-21T05:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.685+0000 7f1ce81cb740 1 -- start start 2024-09-21T05:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.686+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d6b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.686+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.686+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49024/0 (socket says 172.21.15.28:49024) 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1cdad9d640 1 -- 172.21.15.28:0/3594299952 learned_addr learned my addr 172.21.15.28:0/3594299952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1cda59c640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ce1ce06f0 con 0x557ce19c61b0 2024-09-21T05:52:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.687+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557ce1ccfb20 con 0x557ce19a54d0 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557ce1cd3930 con 0x557ce199d2f0 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cda59c640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d6b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cdad9d640 1 -- 172.21.15.28:0/3594299952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 msgr2=0x557ce1d6c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cdad9d640 1 -- 172.21.15.28:0/3594299952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 msgr2=0x557ce1d6b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d6b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cdad9d640 1 -- 172.21.15.28:0/3594299952 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ce1c95530 con 0x557ce199d2f0 2024-09-21T05:52:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.688+0000 7f1cda59c640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:52:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.689+0000 7f1cc3fff640 1 -- 172.21.15.28:0/2117785905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1cd401d020 con 0x7f1cdc1107e0 2024-09-21T05:52:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.690+0000 7f1cc3fff640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 0x7f1cc40b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.690+0000 7f1cda59c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 0x7f1cc40b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.690+0000 7f1cc3fff640 1 -- 172.21.15.28:0/2117785905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1cd40e7400 con 0x7f1cdc1107e0 2024-09-21T05:52:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.691+0000 7f1cda59c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 0x7f1cc40b8d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc06a0f0 tx=0x7f1ccc02a340 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.696+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009c50 tx=0x7f1cd0010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72da455764958fd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.696+0000 7f1cc17fa640 1 -- 172.21.15.28:0/3594299952 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0004030 con 0x557ce199d2f0 2024-09-21T05:52:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.696+0000 7f1cc17fa640 1 -- 172.21.15.28:0/3594299952 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd00041d0 con 0x557ce199d2f0 2024-09-21T05:52:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.696+0000 7f1cc17fa640 1 -- 172.21.15.28:0/3594299952 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00126b0 con 0x557ce199d2f0 2024-09-21T05:52:35.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.698+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 msgr2=0x557ce1d6bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.698+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009c50 tx=0x7f1cd0010740 comp rx=0 tx=0).stop 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 shutdown_connections 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce199d2f0 0x557ce1d6bef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/3594299952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d6b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 >> 172.21.15.28:0/3594299952 conn(0x557ce19ba300 msgr2=0x557ce1d698a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 shutdown_connections 2024-09-21T05:52:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.701+0000 7f1ce81cb740 1 -- 172.21.15.28:0/3594299952 wait complete. 2024-09-21T05:52:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.702+0000 7f1ce81cb740 1 -- start start 2024-09-21T05:52:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.702+0000 7f1ce81cb740 1 Processor -- start 2024-09-21T05:52:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.702+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d775a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1d834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ce1c07ab0 con 0x557ce199d2f0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557ce1c75730 con 0x557ce19a54d0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557ce1d7a130 con 0x557ce19c61b0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33122/0 (socket says 172.21.15.28:33122) 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 -- 172.21.15.28:0/4007685256 learned_addr learned my addr 172.21.15.28:0/4007685256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1ce81cb740 10 client.0 _ll_register_callbacks cb 0x557ce1cc96b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cda59c640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d775a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 -- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 msgr2=0x557ce1d834b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1d834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 -- 172.21.15.28:0/4007685256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 msgr2=0x557ce1d775a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d775a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cdad9d640 1 -- 172.21.15.28:0/4007685256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ce1a31790 con 0x557ce199d2f0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.703+0000 7f1cda59c640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d775a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00045e0 tx=0x7f1cd0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23be5c103e237496 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0003a00 con 0x557ce199d2f0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd00041d0 con 0x557ce199d2f0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1cd0005030 con 0x557ce199d2f0 2024-09-21T05:52:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.704+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1cd00052b0 con 0x557ce199d2f0 2024-09-21T05:52:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.707+0000 7f1ce81cb740 10 client.16866 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.707+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557ce1aaa620 con 0x557ce199d2f0 2024-09-21T05:52:35.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.707+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd00041d0 con 0x557ce199d2f0 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f1cd000f040 con 0x557ce199d2f0 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 10 client.16866 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 10 client.16866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 20 client.16866 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1984d80 0x557ce1d890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 4 client.16866 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ce1ce2640 con 0x557ce1984d80 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.708+0000 7f1cda59c640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1984d80 0x557ce1d890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.709+0000 7f1cda59c640 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1984d80 0x557ce1d890a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d41440d7ab1461f server_cookie=ea71dadb87eb5891 in_seq=0 out_seq=0 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.709+0000 7f1cbeff5640 10 client.16866 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1cbeff5640 1 -- 172.21.15.28:0/4007685256 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557ce1ce2640 con 0x557ce1984d80 2024-09-21T05:52:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1cbeff5640 10 client.16866 handle_command_reply: tid=0 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 client.16866 shutdown 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 msgr2=0x557ce1d77060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00045e0 tx=0x7f1cd0004610 comp rx=0 tx=0).stop 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 shutdown_connections 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1984d80 0x557ce1d890a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1d834b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d775a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1cbeff5640 0 client.16866 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/4007685256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce199d2f0 0x557ce1d77060 secure :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00045e0 tx=0x7f1cd0004610 comp rx=0 tx=0).stop 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 >> 172.21.15.28:0/4007685256 conn(0x557ce19ba300 msgr2=0x557ce1d6c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.712+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 shutdown_connections 2024-09-21T05:52:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.713+0000 7f1ce81cb740 1 -- 172.21.15.28:0/4007685256 wait complete. 2024-09-21T05:52:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.713+0000 7f1ce81cb740 20 client.16866 trim_cache size 0 max 16384 2024-09-21T05:52:35.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.735+0000 7f1ce81cb740 1 Processor -- start 2024-09-21T05:52:35.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.735+0000 7f1ce81cb740 1 -- start start 2024-09-21T05:52:35.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.735+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.735+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce195a1c0 0x557ce1d74230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38274/0 (socket says 172.21.15.28:38274) 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 -- 172.21.15.28:0/153162064 learned_addr learned my addr 172.21.15.28:0/153162064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800e7a0 con 0x557ce19c61b0 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8010310 con 0x557ce19a54d0 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800ee70 con 0x557ce195a1c0 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdb59e640 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce195a1c0 0x557ce1d74230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 -- 172.21.15.28:0/153162064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce195a1c0 msgr2=0x557ce1d74230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce195a1c0 0x557ce1d74230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 -- 172.21.15.28:0/153162064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 msgr2=0x557ce1d833b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.736+0000 7f1cdad9d640 1 -- 172.21.15.28:0/153162064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ce1d6dcb0 con 0x557ce19a54d0 2024-09-21T05:52:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.737+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0004f70 tx=0x7f1cd0005910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d357f5699a055645 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.737+0000 7f1cc0ff9640 1 -- 172.21.15.28:0/153162064 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd003d070 con 0x557ce19a54d0 2024-09-21T05:52:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.737+0000 7f1cc0ff9640 1 -- 172.21.15.28:0/153162064 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd0009530 con 0x557ce19a54d0 2024-09-21T05:52:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.737+0000 7f1cc0ff9640 1 -- 172.21.15.28:0/153162064 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0003800 con 0x557ce19a54d0 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 msgr2=0x557ce1d6d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0004f70 tx=0x7f1cd0005910 comp rx=0 tx=0).stop 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 shutdown_connections 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce195a1c0 0x557ce1d74230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce19c61b0 0x557ce1d833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/153162064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d6d0f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 >> 172.21.15.28:0/153162064 conn(0x557ce19ba300 msgr2=0x557ce1d6c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 shutdown_connections 2024-09-21T05:52:35.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.738+0000 7f1ce81cb740 1 -- 172.21.15.28:0/153162064 wait complete. 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.739+0000 7f1ce81cb740 1 -- start start 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.739+0000 7f1ce81cb740 1 Processor -- start 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d7cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800fc40 con 0x557ce195a1c0 2024-09-21T05:52:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800f540 con 0x557ce19a54d0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800c780 con 0x557ce19c61b0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1ce81cb740 10 client.0 _ll_register_callbacks cb 0x7f1cd4009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cda59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d7cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdb59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdb59e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49048/0 (socket says 172.21.15.28:49048) 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33142/0 (socket says 172.21.15.28:33142) 2024-09-21T05:52:35.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdb59e640 1 -- 172.21.15.28:0/2064098958 learned_addr learned my addr 172.21.15.28:0/2064098958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 -- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 msgr2=0x557ce1df1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 -- 172.21.15.28:0/2064098958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 msgr2=0x557ce1d7cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d7cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.740+0000 7f1cdad9d640 1 -- 172.21.15.28:0/2064098958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ce1df1f80 con 0x557ce195a1c0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.741+0000 7f1cdb59e640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.741+0000 7f1cdad9d640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000c00 tx=0x7f1cd0009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3382ae6ac309f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.741+0000 7f1cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0046060 con 0x557ce195a1c0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd0009530 con 0x557ce195a1c0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1cd00040e0 con 0x557ce195a1c0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1cd003d040 con 0x557ce195a1c0 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1ce81cb740 10 client.16869 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.742+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557ce1df30b0 con 0x557ce195a1c0 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.743+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd0009530 con 0x557ce195a1c0 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.743+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f1cd0040030 con 0x557ce195a1c0 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 10 client.16869 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 20 client.16869 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1df37d0 0x557ce1df5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 4 client.16869 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ce1cde0f0 con 0x557ce1df37d0 2024-09-21T05:52:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.744+0000 7f1cda59c640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1df37d0 0x557ce1df5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.745+0000 7f1cda59c640 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1df37d0 0x557ce1df5bf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c85cbf9564889d server_cookie=c18e843a60df3dcb in_seq=0 out_seq=0 2024-09-21T05:52:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.745+0000 7f1cbeff5640 10 client.16869 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1cbeff5640 1 -- 172.21.15.28:0/2064098958 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ce1cde0f0 con 0x557ce1df37d0 2024-09-21T05:52:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1cbeff5640 10 client.16869 handle_command_reply: tid=0 2024-09-21T05:52:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 client.16869 shutdown 2024-09-21T05:52:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 msgr2=0x557ce1d7c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000c00 tx=0x7f1cd0009f10 comp rx=0 tx=0).stop 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 shutdown_connections 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557ce1df37d0 0x557ce1df5bf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557ce19c61b0 0x557ce1df1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557ce19a54d0 0x557ce1d7cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 --2- 172.21.15.28:0/2064098958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ce195a1c0 0x557ce1d7c7c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 >> 172.21.15.28:0/2064098958 conn(0x557ce19ba300 msgr2=0x557ce1d67ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1cbeff5640 0 client.16869 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 shutdown_connections 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.767+0000 7f1ce81cb740 1 -- 172.21.15.28:0/2064098958 wait complete. 2024-09-21T05:52:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce81cb740 20 client.16869 trim_cache size 0 max 16384 2024-09-21T05:52:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:52:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:52:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:52:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 msgr2=0x7f1cc40b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 0x7f1cc40b8d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc06a0f0 tx=0x7f1ccc02a340 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 msgr2=0x7f1cdc1a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 0x7f1cdc1a20f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400efc0 tx=0x7f1cd400c800 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 shutdown_connections 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1cc40b6850 0x7f1cc40b8d60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc1107e0 0x7f1cdc1a20f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cdc10bf70 0x7f1cdc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.768+0000 7f1ce125c640 1 --2- 172.21.15.28:0/2117785905 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cdc105cf0 0x7f1cdc10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.769+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 >> 172.21.15.28:0/2117785905 conn(0x7f1cdc1016f0 msgr2=0x7f1cdc103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.769+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 shutdown_connections 2024-09-21T05:52:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:35.769+0000 7f1ce125c640 1 -- 172.21.15.28:0/2117785905 wait complete. 2024-09-21T05:52:36.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 -- 172.21.15.28:0/3788038858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e01094c0 msgr2=0x7f87e01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 --2- 172.21.15.28:0/3788038858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e01094c0 0x7f87e01098d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009900 tx=0x7f87d402a410 comp rx=0 tx=0).stop 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 -- 172.21.15.28:0/3788038858 shutdown_connections 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 --2- 172.21.15.28:0/3788038858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e0110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 --2- 172.21.15.28:0/3788038858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e01094c0 0x7f87e01098d0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 --2- 172.21.15.28:0/3788038858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e0103b80 0x7f87e0103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 -- 172.21.15.28:0/3788038858 >> 172.21.15.28:0/3788038858 conn(0x7f87e00ff610 msgr2=0x7f87e0101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.927+0000 7f87e535f640 1 -- 172.21.15.28:0/3788038858 shutdown_connections 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.928+0000 7f87e535f640 1 -- 172.21.15.28:0/3788038858 wait complete. 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.928+0000 7f87e535f640 1 Processor -- start 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.928+0000 7f87e535f640 1 -- start start 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e01094c0 0x7f87e019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e019fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0113ad0 con 0x7f87e0103b80 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0113950 con 0x7f87e01094c0 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87e535f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0113c50 con 0x7f87e0109e10 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33164/0 (socket says 172.21.15.28:33164) 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 -- 172.21.15.28:0/1067639672 learned_addr learned my addr 172.21.15.28:0/1067639672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87de7fc640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e01094c0 0x7f87e019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87df7fe640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e019fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 -- 172.21.15.28:0/1067639672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 msgr2=0x7f87e019fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e019fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 -- 172.21.15.28:0/1067639672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e01094c0 msgr2=0x7f87e019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e01094c0 0x7f87e019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 -- 172.21.15.28:0/1067639672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e01a0450 con 0x7f87e0103b80 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87df7fe640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e019fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87deffd640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f87d000e9f0 tx=0x7f87d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.929+0000 7f87c7fff640 1 -- 172.21.15.28:0/1067639672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d000cd80 con 0x7f87e0103b80 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.930+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e01a0740 con 0x7f87e0103b80 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.930+0000 7f87c7fff640 1 -- 172.21.15.28:0/1067639672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d00108e0 con 0x7f87e0103b80 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.930+0000 7f87c7fff640 1 -- 172.21.15.28:0/1067639672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0018980 con 0x7f87e0103b80 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.930+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e01a6ea0 con 0x7f87e0103b80 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 Processor -- start 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 -- start start 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14226800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14226d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d1419cea0 con 0x557d13e561b0 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d1418c2d0 con 0x557d13e5e390 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d141900e0 con 0x557d13e7f070 2024-09-21T05:52:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87c7fff640 1 -- 172.21.15.28:0/1067639672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f87d0018b20 con 0x7f87e0103b80 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49078/0 (socket says 172.21.15.28:49078) 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87df7fe640 1 -- 172.21.15.28:0/2343371896 learned_addr learned my addr 172.21.15.28:0/2343371896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.933+0000 7f87deffd640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14226800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87de7fc640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14226d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 -- 172.21.15.28:0/2343371896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 msgr2=0x557d14226d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14226d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 -- 172.21.15.28:0/2343371896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 msgr2=0x557d14226800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14226800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 -- 172.21.15.28:0/2343371896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d13eea650 con 0x557d13e7f070 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87deffd640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14226800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87df7fe640 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009fd0 tx=0x7f87d80108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b53a51023a52794f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87c57fa640 1 -- 172.21.15.28:0/2343371896 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8011760 con 0x557d13e7f070 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87c7fff640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 0x7f87c80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87c57fa640 1 -- 172.21.15.28:0/2343371896 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d8011db0 con 0x557d13e7f070 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87c57fa640 1 -- 172.21.15.28:0/2343371896 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d800e3c0 con 0x557d13e7f070 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.934+0000 7f87deffd640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 0x7f87c80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87c7fff640 1 -- 172.21.15.28:0/1067639672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f87d00e7af0 con 0x7f87e0103b80 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2343371896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 msgr2=0x557d142262c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87deffd640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 0x7f87c80b8b90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f87d00e5310 tx=0x7f87d000cf20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009fd0 tx=0x7f87d80108e0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2343371896 shutdown_connections 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14226d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14226800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2343371896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e7f070 0x557d142262c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.935+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2343371896 >> 172.21.15.28:0/2343371896 conn(0x557d13e73460 msgr2=0x557d142241b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2343371896 shutdown_connections 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2343371896 wait complete. 2024-09-21T05:52:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- start start 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 Processor -- start 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e561b0 0x557d14231a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14231f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d13fed700 con 0x557d13e7f070 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d14127160 con 0x557d13e5e390 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d14238e50 con 0x557d13e561b0 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.936+0000 7f87eb8d3740 10 client.0 _ll_register_callbacks cb 0x557d141409e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33184/0 (socket says 172.21.15.28:33184) 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 -- 172.21.15.28:0/4222455296 learned_addr learned my addr 172.21.15.28:0/4222455296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e561b0 0x557d14231a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87de7fc640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14231f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 -- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e561b0 msgr2=0x557d14231a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e561b0 0x557d14231a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 -- 172.21.15.28:0/4222455296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 msgr2=0x557d14231f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14231f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 -- 172.21.15.28:0/4222455296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d14154c90 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87df7fe640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x557d14228760 tx=0x7f87d801bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366a64432c104121 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8020020 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f87d8011ba0 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d8011db0 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f87d8033050 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.937+0000 7f87eb8d3740 10 client.16881 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557d14244bb0 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d8011db0 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f87d800e3c0 con 0x557d13e7f070 2024-09-21T05:52:36.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 10 client.16881 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 10 client.16881 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 20 client.16881 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e3d9f0 0x557d14247f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 4 client.16881 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d1419edf0 con 0x557d13e3d9f0 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.938+0000 7f87de7fc640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e3d9f0 0x557d14247f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.939+0000 7f87c2ff5640 10 client.16881 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.939+0000 7f87de7fc640 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e3d9f0 0x557d14247f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=276ac86e896f272c server_cookie=6643ae1cb1aa9ae1 in_seq=0 out_seq=0 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.939+0000 7f87c2ff5640 1 -- 172.21.15.28:0/4222455296 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d1419edf0 con 0x557d13e3d9f0 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.939+0000 7f87c2ff5640 10 client.16881 handle_command_reply: tid=0 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 client.16881 shutdown 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 msgr2=0x557d14242130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x557d14228760 tx=0x7f87d801bb00 comp rx=0 tx=0).stop 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 shutdown_connections 2024-09-21T05:52:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e3d9f0 0x557d14247f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87c2ff5640 0 client.16881 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e7f070 0x557d14242130 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e5e390 0x557d14231f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/4222455296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e561b0 0x557d14231a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 >> 172.21.15.28:0/4222455296 conn(0x557d13e73460 msgr2=0x557d14229450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 shutdown_connections 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 1 -- 172.21.15.28:0/4222455296 wait complete. 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.940+0000 7f87eb8d3740 20 client.16881 trim_cache size 0 max 16384 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 Processor -- start 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 -- start start 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d14233900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142420e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01ad5c0 con 0x557d13e561b0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01af130 con 0x557d13e73460 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87eb8d3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01adc90 con 0x557d13e5e390 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33200/0 (socket says 172.21.15.28:33200) 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.953+0000 7f87de7fc640 1 -- 172.21.15.28:0/2788548835 learned_addr learned my addr 172.21.15.28:0/2788548835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87df7fe640 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d14233900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87df7fe640 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 -- 172.21.15.28:0/2788548835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 msgr2=0x557d142420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142420e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 -- 172.21.15.28:0/2788548835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 msgr2=0x557d14233900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d14233900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 -- 172.21.15.28:0/2788548835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d142345d0 con 0x557d13e561b0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87de7fc640 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000c00 tx=0x7f87d4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75a024def3653fe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87c4ff9640 1 -- 172.21.15.28:0/2788548835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4035040 con 0x557d13e561b0 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87c4ff9640 1 -- 172.21.15.28:0/2788548835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d40054f0 con 0x557d13e561b0 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.954+0000 7f87c4ff9640 1 -- 172.21.15.28:0/2788548835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4004cb0 con 0x557d13e561b0 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.955+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2788548835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 msgr2=0x557d14227a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.955+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000c00 tx=0x7f87d4009560 comp rx=0 tx=0).stop 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2788548835 shutdown_connections 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d14227a00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/2788548835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d14233900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2788548835 >> 172.21.15.28:0/2788548835 conn(0x557d142320e0 msgr2=0x557d142272f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.956+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2788548835 shutdown_connections 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.957+0000 7f87eb8d3740 1 -- 172.21.15.28:0/2788548835 wait complete. 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.957+0000 7f87eb8d3740 1 -- start start 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.958+0000 7f87eb8d3740 1 Processor -- start 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.958+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87eb8d3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33208/0 (socket says 172.21.15.28:33208) 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87df7fe640 1 -- 172.21.15.28:0/3039868137 learned_addr learned my addr 172.21.15.28:0/3039868137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d142b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01aea60 con 0x557d13e561b0 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01ae360 con 0x557d13e73460 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01ab5a0 con 0x557d13e5e390 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.959+0000 7f87df7fe640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d142b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 -- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 msgr2=0x557d142a6d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142a6d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 -- 172.21.15.28:0/3039868137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 msgr2=0x557d142b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d142b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d142b1c90 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87df7fe640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f87d8002230 tx=0x7f87d8004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f58a3406b3e17e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.960+0000 7f87c2ff5640 1 -- 172.21.15.28:0/3039868137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8020040 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.961+0000 7f87eb8d3740 10 client.0 _ll_register_callbacks cb 0x7f87d8009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.961+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d40054f0 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.961+0000 7f87c2ff5640 1 -- 172.21.15.28:0/3039868137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f87d800e3c0 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87c2ff5640 1 -- 172.21.15.28:0/3039868137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f87d8010e90 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 10 client.16887 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=41}) -- 0x557d142b2d40 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87c2ff5640 1 -- 172.21.15.28:0/3039868137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f87d80119a0 con 0x557d13e561b0 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 10 client.16887 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 10 client.16887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 20 client.16887 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e7f070 0x557d142b5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 4 client.16887 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.962+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d1419a8a0 con 0x557d13e7f070 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.963+0000 7f87de7fc640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e7f070 0x557d142b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.963+0000 7f87de7fc640 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e7f070 0x557d142b5410 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee9c0c3f5363cfbf server_cookie=cdc9edb35104f3f7 in_seq=0 out_seq=0 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.963+0000 7f87c2ff5640 10 client.16887 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.965+0000 7f87c2ff5640 1 -- 172.21.15.28:0/3039868137 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d1419a8a0 con 0x557d13e7f070 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.965+0000 7f87c2ff5640 10 client.16887 handle_command_reply: tid=0 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.965+0000 7f87eb8d3740 1 client.16887 shutdown 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.965+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 msgr2=0x557d142a6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.965+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f87d8002230 tx=0x7f87d8004d50 comp rx=0 tx=0).stop 2024-09-21T05:52:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 shutdown_connections 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d13e7f070 0x557d142b5410 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d13e73460 0x557d142b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d13e5e390 0x557d142a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87c2ff5640 0 client.16887 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 --2- 172.21.15.28:0/3039868137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d13e561b0 0x557d142a6820 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 >> 172.21.15.28:0/3039868137 conn(0x557d142320e0 msgr2=0x557d14227c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 shutdown_connections 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 1 -- 172.21.15.28:0/3039868137 wait complete. 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.966+0000 7f87eb8d3740 20 client.16887 trim_cache size 0 max 16384 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 msgr2=0x7f87c80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 0x7f87c80b8b90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f87d00e5310 tx=0x7f87d000cf20 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 msgr2=0x7f87e019a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f87d000e9f0 tx=0x7f87d000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 shutdown_connections 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87e0109e10 0x7f87e019fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87e01094c0 0x7f87e019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87c80b6680 0x7f87c80b8b90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 --2- 172.21.15.28:0/1067639672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e0103b80 0x7f87e019a4e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.967+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 >> 172.21.15.28:0/1067639672 conn(0x7f87e00ff610 msgr2=0x7f87e0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.968+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 shutdown_connections 2024-09-21T05:52:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:35.968+0000 7f87e535f640 1 -- 172.21.15.28:0/1067639672 wait complete. 2024-09-21T05:52:36.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:36 smithi028 ceph-mon[31227]: pgmap v492: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 4.66k op/s 2024-09-21T05:52:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:36 smithi129 ceph-mon[34885]: pgmap v492: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 4.66k op/s 2024-09-21T05:52:36.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:36 smithi181 ceph-mon[34819]: pgmap v492: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 4.66k op/s 2024-09-21T05:52:37.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:52:37.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:38.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.414+0000 7f1e4b756640 1 -- 172.21.15.28:0/20098546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 msgr2=0x7f1e4410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.414+0000 7f1e4b756640 1 --2- 172.21.15.28:0/20098546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e4410a3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009860 tx=0x7f1e4002a380 comp rx=0 tx=0).stop 2024-09-21T05:52:38.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.417+0000 7f1e4b756640 1 -- 172.21.15.28:0/20098546 shutdown_connections 2024-09-21T05:52:38.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.417+0000 7f1e4b756640 1 --2- 172.21.15.28:0/20098546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4411cc30 0x7f1e4411f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.417+0000 7f1e4b756640 1 --2- 172.21.15.28:0/20098546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4410a9e0 0x7f1e4410adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.417+0000 7f1e4b756640 1 --2- 172.21.15.28:0/20098546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e4410a3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.417+0000 7f1e4b756640 1 -- 172.21.15.28:0/20098546 >> 172.21.15.28:0/20098546 conn(0x7f1e44070ec0 msgr2=0x7f1e440712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.420+0000 7f1e4b756640 1 -- 172.21.15.28:0/20098546 shutdown_connections 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.420+0000 7f1e4b756640 1 -- 172.21.15.28:0/20098546 wait complete. 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 Processor -- start 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 -- start start 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4410a9e0 0x7f1e4411c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e441186a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 0x7f1e44118be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44114bc0 con 0x7f1e4411cc30 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44114a40 con 0x7f1e4410a9e0 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.421+0000 7f1e4b756640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44114d40 con 0x7f1e44111f40 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.422+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 0x7f1e44118be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.422+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 0x7f1e44118be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33230/0 (socket says 172.21.15.28:33230) 2024-09-21T05:52:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.422+0000 7f1e49cf3640 1 -- 172.21.15.28:0/1555668441 learned_addr learned my addr 172.21.15.28:0/1555668441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.422+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e441186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.422+0000 7f1e494f2640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4410a9e0 0x7f1e4411c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.423+0000 7f1e48cf1640 1 -- 172.21.15.28:0/1555668441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4410a9e0 msgr2=0x7f1e4411c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.423+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4410a9e0 0x7f1e4411c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.423+0000 7f1e48cf1640 1 -- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 msgr2=0x7f1e44118be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.423+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 0x7f1e44118be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.423+0000 7f1e48cf1640 1 -- 172.21.15.28:0/1555668441 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e441193e0 con 0x7f1e44111f40 2024-09-21T05:52:38.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.425+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e441186a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e4406c6e0 tx=0x7f1e3401fe40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.427+0000 7f1e3a7fc640 1 -- 172.21.15.28:0/1555668441 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3401db60 con 0x7f1e44111f40 2024-09-21T05:52:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.428+0000 7f1e3a7fc640 1 -- 172.21.15.28:0/1555668441 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3401dd00 con 0x7f1e44111f40 2024-09-21T05:52:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.429+0000 7f1e3a7fc640 1 -- 172.21.15.28:0/1555668441 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34003dd0 con 0x7f1e44111f40 2024-09-21T05:52:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.430+0000 7f1e4b756640 1 -- 172.21.15.28:0/1555668441 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4407b680 con 0x7f1e44111f40 2024-09-21T05:52:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.430+0000 7f1e4b756640 1 -- 172.21.15.28:0/1555668441 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4407bb40 con 0x7f1e44111f40 2024-09-21T05:52:38.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 Processor -- start 2024-09-21T05:52:38.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 -- start start 2024-09-21T05:52:38.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1e61b0 0x55741d5b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55741d52cea0 con 0x55741d1ee390 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55741d51c2d0 con 0x55741d1e61b0 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e51cca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55741d5200e0 con 0x55741d20f070 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.434+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49134/0 (socket says 172.21.15.28:49134) 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 -- 172.21.15.28:0/4167908936 learned_addr learned my addr 172.21.15.28:0/4167908936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 -- 172.21.15.28:0/4167908936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1e61b0 msgr2=0x55741d5b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1e61b0 0x55741d5b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 -- 172.21.15.28:0/4167908936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 msgr2=0x55741d5b6b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5b6b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.435+0000 7f1e49cf3640 1 -- 172.21.15.28:0/4167908936 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741d27a650 con 0x55741d20f070 2024-09-21T05:52:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.437+0000 7f1e3a7fc640 1 -- 172.21.15.28:0/1555668441 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1e34003f70 con 0x7f1e44111f40 2024-09-21T05:52:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.438+0000 7f1e3a7fc640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 0x7f1e300b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.439+0000 7f1e3a7fc640 1 -- 172.21.15.28:0/1555668441 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1e340f88f0 con 0x7f1e44111f40 2024-09-21T05:52:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.440+0000 7f1e494f2640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 0x7f1e300b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.442+0000 7f1e494f2640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 0x7f1e300b8bc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1e4411c110 tx=0x7f1e40000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.445+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0114c0 tx=0x7f1e3c011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6815580702747a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.448+0000 7f1e4b756640 1 -- 172.21.15.28:0/4167908936 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c004b20 con 0x55741d20f070 2024-09-21T05:52:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.448+0000 7f1e4b756640 1 -- 172.21.15.28:0/4167908936 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3c004cc0 con 0x55741d20f070 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 -- 172.21.15.28:0/4167908936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 msgr2=0x55741d5b8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0114c0 tx=0x7f1e3c011980 comp rx=0 tx=0).stop 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 -- 172.21.15.28:0/4167908936 shutdown_connections 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1e61b0 0x55741d5b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 --2- 172.21.15.28:0/4167908936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5b8f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.449+0000 7f1e51cca740 1 -- 172.21.15.28:0/4167908936 >> 172.21.15.28:0/4167908936 conn(0x55741d203460 msgr2=0x55741d5b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.453+0000 7f1e51cca740 1 -- 172.21.15.28:0/4167908936 shutdown_connections 2024-09-21T05:52:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.454+0000 7f1e51cca740 1 -- 172.21.15.28:0/4167908936 wait complete. 2024-09-21T05:52:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.454+0000 7f1e51cca740 1 -- start start 2024-09-21T05:52:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.454+0000 7f1e51cca740 1 Processor -- start 2024-09-21T05:52:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 0x55741d5c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55741d37d700 con 0x55741d1ee390 2024-09-21T05:52:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55741d4b7160 con 0x55741d20f070 2024-09-21T05:52:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55741d5c4a20 con 0x55741d1e61b0 2024-09-21T05:52:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.455+0000 7f1e51cca740 10 client.0 _ll_register_callbacks cb 0x55741d4d09e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.456+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.458+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38360/0 (socket says 172.21.15.28:38360) 2024-09-21T05:52:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.458+0000 7f1e49cf3640 1 -- 172.21.15.28:0/904081051 learned_addr learned my addr 172.21.15.28:0/904081051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.459+0000 7f1e494f2640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 0x55741d5c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.460+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 -- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 msgr2=0x55741d5c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 -- 172.21.15.28:0/904081051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 msgr2=0x55741d5c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 -- 172.21.15.28:0/904081051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741d4e4c90 con 0x55741d1e61b0 2024-09-21T05:52:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.461+0000 7f1e494f2640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 0x55741d5c1b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e40035040 tx=0x7f1e4002a380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a4b87c8407bc7c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.462+0000 7f1e2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.465+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40045040 con 0x55741d1e61b0 2024-09-21T05:52:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.465+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e3c004cc0 con 0x55741d1e61b0 2024-09-21T05:52:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.465+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1e40005040 con 0x55741d1e61b0 2024-09-21T05:52:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.466+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1e40057050 con 0x55741d1e61b0 2024-09-21T05:52:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.469+0000 7f1e51cca740 10 client.26485 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.469+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55741d5d0730 con 0x55741d1e61b0 2024-09-21T05:52:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.472+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3c004cc0 con 0x55741d1e61b0 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.473+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f1e400035f0 con 0x55741d1e61b0 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 10 client.26485 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 10 client.26485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 20 client.26485 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d1cd9f0 0x55741d5d3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 4 client.26485 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.475+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55741d52edf0 con 0x55741d1cd9f0 2024-09-21T05:52:38.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.478+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d1cd9f0 0x55741d5d3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.480+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d1cd9f0 0x55741d5d3c00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6d1dbc03c1fcfdd server_cookie=b3c32548527fa60a in_seq=0 out_seq=0 2024-09-21T05:52:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.481+0000 7f1e2cff9640 10 client.26485 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:52:38.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.484+0000 7f1e2cff9640 1 -- 172.21.15.28:0/904081051 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55741d52edf0 con 0x55741d1cd9f0 2024-09-21T05:52:38.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.484+0000 7f1e2cff9640 10 client.26485 handle_command_reply: tid=0 2024-09-21T05:52:38.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:38 smithi028 ceph-mon[31227]: pgmap v493: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 28 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.484+0000 7f1e51cca740 1 client.26485 shutdown 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 msgr2=0x55741d5c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 0x55741d5c1b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e40035040 tx=0x7f1e4002a380 comp rx=0 tx=0).stop 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 shutdown_connections 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d1cd9f0 0x55741d5d3c00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 --2- 172.21.15.28:0/904081051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1e61b0 0x55741d5c1b50 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e40035040 tx=0x7f1e4002a380 comp rx=0 tx=0).stop 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 >> 172.21.15.28:0/904081051 conn(0x55741d203460 msgr2=0x55741d5b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.485+0000 7f1e2cff9640 0 client.26485 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:52:38.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.486+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 shutdown_connections 2024-09-21T05:52:38.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.487+0000 7f1e51cca740 1 -- 172.21.15.28:0/904081051 wait complete. 2024-09-21T05:52:38.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.488+0000 7f1e51cca740 20 client.26485 trim_cache size 0 max 16384 2024-09-21T05:52:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.519+0000 7f1e51cca740 1 Processor -- start 2024-09-21T05:52:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e51cca740 1 -- start start 2024-09-21T05:52:38.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1ee390 0x55741d5c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e48cf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e48cf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38376/0 (socket says 172.21.15.28:38376) 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.520+0000 7f1e49cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33236/0 (socket says 172.21.15.28:33236) 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.521+0000 7f1e48cf1640 1 -- 172.21.15.28:0/1212175714 learned_addr learned my addr 172.21.15.28:0/1212175714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.521+0000 7f1e494f2640 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1ee390 0x55741d5c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.521+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44084190 con 0x55741d1a2da0 2024-09-21T05:52:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.521+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44085d00 con 0x55741d20f070 2024-09-21T05:52:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.521+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44084860 con 0x55741d1ee390 2024-09-21T05:52:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.522+0000 7f1e49cf3640 1 -- 172.21.15.28:0/1212175714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1ee390 msgr2=0x55741d5c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.522+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1ee390 0x55741d5c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.522+0000 7f1e49cf3640 1 -- 172.21.15.28:0/1212175714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 msgr2=0x55741d5b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.522+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.522+0000 7f1e49cf3640 1 -- 172.21.15.28:0/1212175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741d1cdc20 con 0x55741d1a2da0 2024-09-21T05:52:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.523+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009700 tx=0x7f1e3c005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fc6ac64f9cd74c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.524+0000 7f1e2effd640 1 -- 172.21.15.28:0/1212175714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c028070 con 0x55741d1a2da0 2024-09-21T05:52:38.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.524+0000 7f1e2effd640 1 -- 172.21.15.28:0/1212175714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3c005990 con 0x55741d1a2da0 2024-09-21T05:52:38.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.525+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 msgr2=0x55741d5bed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.525+0000 7f1e51cca740 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009700 tx=0x7f1e3c005820 comp rx=0 tx=0).stop 2024-09-21T05:52:38.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.531+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 shutdown_connections 2024-09-21T05:52:38.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.531+0000 7f1e51cca740 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1a2da0 0x55741d5bed40 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.531+0000 7f1e51cca740 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d20f070 0x55741d5b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.531+0000 7f1e51cca740 1 --2- 172.21.15.28:0/1212175714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d1ee390 0x55741d5c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.531+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 >> 172.21.15.28:0/1212175714 conn(0x55741d203460 msgr2=0x55741d5b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.535+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 shutdown_connections 2024-09-21T05:52:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.535+0000 7f1e51cca740 1 -- 172.21.15.28:0/1212175714 wait complete. 2024-09-21T05:52:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.535+0000 7f1e51cca740 1 -- start start 2024-09-21T05:52:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.535+0000 7f1e51cca740 1 Processor -- start 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44085630 con 0x55741d1ee390 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44084f30 con 0x55741d1a2da0 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44082220 con 0x55741d20f070 2024-09-21T05:52:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.536+0000 7f1e51cca740 10 client.0 _ll_register_callbacks cb 0x7f1e3c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38392/0 (socket says 172.21.15.28:38392) 2024-09-21T05:52:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 -- 172.21.15.28:0/3532024764 learned_addr learned my addr 172.21.15.28:0/3532024764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e49cf3640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 -- 172.21.15.28:0/3532024764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 msgr2=0x55741d5c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 -- 172.21.15.28:0/3532024764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 msgr2=0x55741d5c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.538+0000 7f1e494f2640 1 -- 172.21.15.28:0/3532024764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741d63d820 con 0x55741d1a2da0 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.539+0000 7f1e494f2640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1e4002a970 tx=0x7f1e400058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46c3277bd19eeb61 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.539+0000 7f1e2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.539+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40045040 con 0x55741d1a2da0 2024-09-21T05:52:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.539+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f1e40005040 con 0x55741d1a2da0 2024-09-21T05:52:38.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.543+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e3c005990 con 0x55741d1a2da0 2024-09-21T05:52:38.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.547+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1e40037040 con 0x55741d1a2da0 2024-09-21T05:52:38.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.547+0000 7f1e51cca740 10 client.26486 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:38.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.547+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55741d63eaa0 con 0x55741d1a2da0 2024-09-21T05:52:38.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e3c005990 con 0x55741d1a2da0 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f1e40004a40 con 0x55741d1a2da0 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 10 client.26486 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 10 client.26486 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 20 client.26486 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d63f220 0x55741d641640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 4 client.26486 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:52:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.548+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55741d52a8a0 con 0x55741d63f220 2024-09-21T05:52:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.558+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d63f220 0x55741d641640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.575+0000 7f1e2cff9640 10 client.26486 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:52:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.575+0000 7f1e48cf1640 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d63f220 0x55741d641640 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ad496d60aa47fa0 server_cookie=de2c9e8f0e0cb85e in_seq=0 out_seq=0 2024-09-21T05:52:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.575+0000 7f1e2cff9640 1 -- 172.21.15.28:0/3532024764 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55741d52a8a0 con 0x55741d63f220 2024-09-21T05:52:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.575+0000 7f1e2cff9640 10 client.26486 handle_command_reply: tid=0 2024-09-21T05:52:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.575+0000 7f1e51cca740 1 client.26486 shutdown 2024-09-21T05:52:38.582 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:52:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 msgr2=0x55741d5c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1e4002a970 tx=0x7f1e400058c0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 shutdown_connections 2024-09-21T05:52:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55741d63f220 0x55741d641640 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55741d20f070 0x55741d5c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55741d1ee390 0x55741d5c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 --2- 172.21.15.28:0/3532024764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55741d1a2da0 0x55741d5c7f90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 >> 172.21.15.28:0/3532024764 conn(0x55741d203460 msgr2=0x55741d5b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e2cff9640 0 client.26486 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 shutdown_connections 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.579+0000 7f1e51cca740 1 -- 172.21.15.28:0/3532024764 wait complete. 2024-09-21T05:52:38.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.580+0000 7f1e51cca740 20 client.26486 trim_cache size 0 max 16384 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.581+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 msgr2=0x7f1e300b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.581+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 0x7f1e300b8bc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1e4411c110 tx=0x7f1e40000bc0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.581+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 msgr2=0x7f1e441186a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.581+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e441186a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e4406c6e0 tx=0x7f1e3401fe40 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.582+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 shutdown_connections 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.582+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1e300b66b0 0x7f1e300b8bc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.582+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e4411cc30 0x7f1e44118be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.582+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e44111f40 0x7f1e441186a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.582+0000 7f1e2ffff640 1 --2- 172.21.15.28:0/1555668441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4410a9e0 0x7f1e4411c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.583+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 >> 172.21.15.28:0/1555668441 conn(0x7f1e44070ec0 msgr2=0x7f1e4410ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.583+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 shutdown_connections 2024-09-21T05:52:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:38.583+0000 7f1e2ffff640 1 -- 172.21.15.28:0/1555668441 wait complete. 2024-09-21T05:52:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:38 smithi129 ceph-mon[34885]: pgmap v493: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 28 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-09-21T05:52:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:38 smithi181 ceph-mon[34819]: pgmap v493: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 28 KiB/s rd, 19 MiB/s wr, 4.44k op/s 2024-09-21T05:52:39.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:52:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:40.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:40 smithi028 ceph-mon[31227]: pgmap v494: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 32 KiB/s rd, 22 MiB/s wr, 5.05k op/s 2024-09-21T05:52:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:40 smithi129 ceph-mon[34885]: pgmap v494: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 32 KiB/s rd, 22 MiB/s wr, 5.05k op/s 2024-09-21T05:52:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 -- 172.21.15.28:0/162168029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 msgr2=0x7f44d01087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 --2- 172.21.15.28:0/162168029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d01087b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009900 tx=0x7f44c402a410 comp rx=0 tx=0).stop 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 -- 172.21.15.28:0/162168029 shutdown_connections 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 --2- 172.21.15.28:0/162168029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44d0108cf0 0x7f44d010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 --2- 172.21.15.28:0/162168029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d01087b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 --2- 172.21.15.28:0/162168029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0102a60 0x7f44d0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.698+0000 7f44d7320640 1 -- 172.21.15.28:0/162168029 >> 172.21.15.28:0/162168029 conn(0x7f44d0079730 msgr2=0x7f44d0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.699+0000 7f44d7320640 1 -- 172.21.15.28:0/162168029 shutdown_connections 2024-09-21T05:52:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.699+0000 7f44d7320640 1 -- 172.21.15.28:0/162168029 wait complete. 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 Processor -- start 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 -- start start 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44d0102a60 0x7f44d019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0108cf0 0x7f44d01a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0111890 con 0x7f44d01083a0 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0111710 con 0x7f44d0102a60 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.700+0000 7f44d7320640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0111a10 con 0x7f44d0108cf0 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33256/0 (socket says 172.21.15.28:33256) 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 -- 172.21.15.28:0/2199564172 learned_addr learned my addr 172.21.15.28:0/2199564172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d58bd640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0108cf0 0x7f44d01a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 -- 172.21.15.28:0/2199564172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0108cf0 msgr2=0x7f44d01a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0108cf0 0x7f44d01a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 -- 172.21.15.28:0/2199564172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44d0102a60 msgr2=0x7f44d019c740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44d0102a60 0x7f44d019c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.701+0000 7f44d48bb640 1 -- 172.21.15.28:0/2199564172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d01a26b0 con 0x7f44d01083a0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44d48bb640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f44c4005940 tx=0x7f44c4004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44ce7fc640 1 -- 172.21.15.28:0/2199564172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c403f030 con 0x7f44d01083a0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44ce7fc640 1 -- 172.21.15.28:0/2199564172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c40054b0 con 0x7f44d01083a0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44ce7fc640 1 -- 172.21.15.28:0/2199564172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4035070 con 0x7f44d01083a0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d01a2940 con 0x7f44d01083a0 2024-09-21T05:52:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.702+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d01a9100 con 0x7f44d01083a0 2024-09-21T05:52:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.706+0000 7f44dd894740 1 Processor -- start 2024-09-21T05:52:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.706+0000 7f44dd894740 1 -- start start 2024-09-21T05:52:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.706+0000 7f44dd894740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.706+0000 7f44dd894740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a421b0 0x5611b9e12be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44dd894740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e13120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44dd894740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611b9d88ea0 con 0x5611b9a4a390 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44dd894740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5611b9d782d0 con 0x5611b9a421b0 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44dd894740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5611b9d7c0e0 con 0x5611b9a6b070 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49192/0 (socket says 172.21.15.28:49192) 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 -- 172.21.15.28:0/1282409676 learned_addr learned my addr 172.21.15.28:0/1282409676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 -- 172.21.15.28:0/1282409676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a421b0 msgr2=0x5611b9e12be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a421b0 0x5611b9e12be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 -- 172.21.15.28:0/1282409676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 msgr2=0x5611b9e13120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e13120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.707+0000 7f44d58bd640 1 -- 172.21.15.28:0/1282409676 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611b9ad6650 con 0x5611b9a6b070 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.708+0000 7f44d58bd640 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b700 tx=0x7f44c000bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5edda860d9728a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.708+0000 7f44b77fe640 1 -- 172.21.15.28:0/1282409676 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0003d70 con 0x5611b9a6b070 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.708+0000 7f44b77fe640 1 -- 172.21.15.28:0/1282409676 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c0015070 con 0x5611b9a6b070 2024-09-21T05:52:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.708+0000 7f44b77fe640 1 -- 172.21.15.28:0/1282409676 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c00106a0 con 0x5611b9a6b070 2024-09-21T05:52:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.708+0000 7f44ce7fc640 1 -- 172.21.15.28:0/2199564172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f44c4042030 con 0x7f44d01083a0 2024-09-21T05:52:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.709+0000 7f44ce7fc640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 0x7f44b80b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.709+0000 7f44dd894740 1 -- 172.21.15.28:0/1282409676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 msgr2=0x5611b9e126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.709+0000 7f44dd894740 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b700 tx=0x7f44c000bbc0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44dd894740 1 -- 172.21.15.28:0/1282409676 shutdown_connections 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44dd894740 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e13120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44dd894740 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a421b0 0x5611b9e12be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44dd894740 1 --2- 172.21.15.28:0/1282409676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e126a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44dd894740 1 -- 172.21.15.28:0/1282409676 >> 172.21.15.28:0/1282409676 conn(0x5611b9a5f460 msgr2=0x5611b9e10590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.710+0000 7f44ce7fc640 1 -- 172.21.15.28:0/2199564172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44c4107c10 con 0x7f44d01083a0 2024-09-21T05:52:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.711+0000 7f44d50bc640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 0x7f44b80b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.712+0000 7f44dd894740 1 -- 172.21.15.28:0/1282409676 shutdown_connections 2024-09-21T05:52:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.712+0000 7f44dd894740 1 -- 172.21.15.28:0/1282409676 wait complete. 2024-09-21T05:52:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.712+0000 7f44d50bc640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 0x7f44b80b8c90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5611b9e11320 tx=0x7f44c80091f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.713+0000 7f44dd894740 1 -- start start 2024-09-21T05:52:40.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.713+0000 7f44dd894740 1 Processor -- start 2024-09-21T05:52:40.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44dd894740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e2ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44dd894740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33258/0 (socket says 172.21.15.28:33258) 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44d50bc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e2ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44d50bc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e2ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49204/0 (socket says 172.21.15.28:49204) 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.714+0000 7f44dd894740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a6b070 0x5611b9e3b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44d48bb640 1 -- 172.21.15.28:0/1903478476 learned_addr learned my addr 172.21.15.28:0/1903478476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611b9bd9700 con 0x5611b9a4a390 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5611b9d13160 con 0x5611b9a6b070 2024-09-21T05:52:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5611b9e31de0 con 0x5611b9a421b0 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44d58bd640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a6b070 0x5611b9e3b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.715+0000 7f44dd894740 10 client.0 _ll_register_callbacks cb 0x5611b9d2c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 -- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 msgr2=0x5611b9e2ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e2ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 -- 172.21.15.28:0/1903478476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a6b070 msgr2=0x5611b9e3b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a6b070 0x5611b9e3b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611b9d40c90 con 0x5611b9a4a390 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44c40cd9b0 tx=0x7f44c40cd8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee148717d3141b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.716+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c0015070 con 0x5611b9a4a390 2024-09-21T05:52:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.717+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1903478476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4049040 con 0x5611b9a4a390 2024-09-21T05:52:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.717+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1903478476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44c40d2180 con 0x5611b9a4a390 2024-09-21T05:52:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.717+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1903478476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f44c40ce120 con 0x5611b9a4a390 2024-09-21T05:52:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.717+0000 7f44dd894740 10 client.16908 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.717+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=41}) -- 0x5611b9e3d9a0 con 0x5611b9a4a390 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1903478476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f44c40c9c90 con 0x5611b9a4a390 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 10 client.16908 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 20 client.16908 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b9a299f0 0x5611b9e40e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 4 client.16908 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.718+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f44c0015070 con 0x5611b9a299f0 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.719+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b9a299f0 0x5611b9e40e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.719+0000 7f44b4ff9640 10 client.16908 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.719+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b9a299f0 0x5611b9e40e00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5268c759721a9a server_cookie=4df20fb56603c184 in_seq=0 out_seq=0 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.720+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1903478476 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f44c0015070 con 0x5611b9a299f0 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.720+0000 7f44b4ff9640 10 client.16908 handle_command_reply: tid=0 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.720+0000 7f44dd894740 1 client.16908 shutdown 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.720+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 msgr2=0x5611b9e2f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.720+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44c40cd9b0 tx=0x7f44c40cd8b0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 shutdown_connections 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b9a299f0 0x5611b9e40e00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44b4ff9640 0 client.16908 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a6b070 0x5611b9e3b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a4a390 0x5611b9e2f2a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 --2- 172.21.15.28:0/1903478476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e2ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 >> 172.21.15.28:0/1903478476 conn(0x5611b9a5f460 msgr2=0x5611b9e14040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 shutdown_connections 2024-09-21T05:52:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 1 -- 172.21.15.28:0/1903478476 wait complete. 2024-09-21T05:52:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.721+0000 7f44dd894740 20 client.16908 trim_cache size 0 max 16384 2024-09-21T05:52:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.743+0000 7f44dd894740 1 Processor -- start 2024-09-21T05:52:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.743+0000 7f44dd894740 1 -- start start 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e30990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a6b070 0x5611b9e2c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0075440 con 0x5611b9a6b070 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01aac90 con 0x5611b9a4a390 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.744+0000 7f44dd894740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a97f0 con 0x5611b9a421b0 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.745+0000 7f44d50bc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e30990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.745+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.745+0000 7f44d48bb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38434/0 (socket says 172.21.15.28:38434) 2024-09-21T05:52:40.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.745+0000 7f44d48bb640 1 -- 172.21.15.28:0/1895100331 learned_addr learned my addr 172.21.15.28:0/1895100331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 -- 172.21.15.28:0/1895100331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 msgr2=0x5611b9e30990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e30990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 -- 172.21.15.28:0/1895100331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a6b070 msgr2=0x5611b9e2c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a6b070 0x5611b9e2c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 -- 172.21.15.28:0/1895100331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611b9e30020 con 0x5611b9a4a390 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f44c40f5140 tx=0x7f44c4045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13eaeb7f786aa770 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.746+0000 7f44b6ffd640 1 -- 172.21.15.28:0/1895100331 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4049040 con 0x5611b9a4a390 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.747+0000 7f44b6ffd640 1 -- 172.21.15.28:0/1895100331 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c40d2500 con 0x5611b9a4a390 2024-09-21T05:52:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.747+0000 7f44b6ffd640 1 -- 172.21.15.28:0/1895100331 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c40efa30 con 0x5611b9a4a390 2024-09-21T05:52:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.748+0000 7f44dd894740 1 -- 172.21.15.28:0/1895100331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 msgr2=0x5611b9e2bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.748+0000 7f44dd894740 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f44c40f5140 tx=0x7f44c4045000 comp rx=0 tx=0).stop 2024-09-21T05:52:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 -- 172.21.15.28:0/1895100331 shutdown_connections 2024-09-21T05:52:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a6b070 0x5611b9e2c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e2bf50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 --2- 172.21.15.28:0/1895100331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a421b0 0x5611b9e30990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 -- 172.21.15.28:0/1895100331 >> 172.21.15.28:0/1895100331 conn(0x5611b9a5f460 msgr2=0x5611b9e30280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 -- 172.21.15.28:0/1895100331 shutdown_connections 2024-09-21T05:52:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.749+0000 7f44dd894740 1 -- 172.21.15.28:0/1895100331 wait complete. 2024-09-21T05:52:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.750+0000 7f44dd894740 1 -- start start 2024-09-21T05:52:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.750+0000 7f44dd894740 1 Processor -- start 2024-09-21T05:52:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.751+0000 7f44dd894740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 0x5611b9e33250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.751+0000 7f44dd894740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e33790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.751+0000 7f44dd894740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e33cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44dd894740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01aa5c0 con 0x5611b9a421b0 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44dd894740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a9ec0 con 0x5611b9a4a390 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44d58bd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e33cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44d58bd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e33cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49218/0 (socket says 172.21.15.28:49218) 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44d58bd640 1 -- 172.21.15.28:0/1376785903 learned_addr learned my addr 172.21.15.28:0/1376785903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e33790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0073420 con 0x5611b9a6b070 2024-09-21T05:52:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.752+0000 7f44dd894740 10 client.0 _ll_register_callbacks cb 0x7f44c4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:40.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.753+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 0x5611b9e33250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.754+0000 7f44d50bc640 1 -- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 msgr2=0x5611b9e33cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.754+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e33cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.754+0000 7f44d50bc640 1 -- 172.21.15.28:0/1376785903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 msgr2=0x5611b9e33790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.754+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e33790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.754+0000 7f44d50bc640 1 -- 172.21.15.28:0/1376785903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611b9eaa9c0 con 0x5611b9a421b0 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.755+0000 7f44d50bc640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 0x5611b9e33250 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x5611b9e2f830 tx=0x7f44c8004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e52b33dd5a253be server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.755+0000 7f44b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.756+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c800eb00 con 0x5611b9a421b0 2024-09-21T05:52:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.756+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f44c800eca0 con 0x5611b9a421b0 2024-09-21T05:52:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.757+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c40d2500 con 0x5611b9a421b0 2024-09-21T05:52:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.757+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f44c802c440 con 0x5611b9a421b0 2024-09-21T05:52:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.758+0000 7f44dd894740 10 client.16911 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.758+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5611b9eabc60 con 0x5611b9a421b0 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.758+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c40d2500 con 0x5611b9a421b0 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.758+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f44c802a070 con 0x5611b9a421b0 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 10 client.16911 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 20 client.16911 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b99feda0 0x5611b9eae340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 4 client.16911 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:52:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.759+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611b9d8adf0 con 0x5611b99feda0 2024-09-21T05:52:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.760+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b99feda0 0x5611b9eae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.762+0000 7f44b4ff9640 10 client.16911 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.762+0000 7f44d48bb640 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b99feda0 0x5611b9eae340 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e56f708b58d166a8 server_cookie=bbd081a3fc9f333f in_seq=0 out_seq=0 2024-09-21T05:52:40.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.762+0000 7f44b4ff9640 1 -- 172.21.15.28:0/1376785903 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611b9d8adf0 con 0x5611b99feda0 2024-09-21T05:52:40.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.762+0000 7f44b4ff9640 10 client.16911 handle_command_reply: tid=0 2024-09-21T05:52:40.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.763+0000 7f44dd894740 1 client.16911 shutdown 2024-09-21T05:52:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 msgr2=0x5611b9e33250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 0x5611b9e33250 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x5611b9e2f830 tx=0x7f44c8004b50 comp rx=0 tx=0).stop 2024-09-21T05:52:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 shutdown_connections 2024-09-21T05:52:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5611b99feda0 0x5611b9eae340 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5611b9a6b070 0x5611b9e33cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44b4ff9640 0 client.16911 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611b9a4a390 0x5611b9e33790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 --2- 172.21.15.28:0/1376785903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611b9a421b0 0x5611b9e33250 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.764+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 >> 172.21.15.28:0/1376785903 conn(0x5611b9a5f460 msgr2=0x5611b9e13f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.765+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 shutdown_connections 2024-09-21T05:52:40.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.765+0000 7f44dd894740 1 -- 172.21.15.28:0/1376785903 wait complete. 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.765+0000 7f44dd894740 20 client.16911 trim_cache size 0 max 16384 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.766+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 msgr2=0x7f44b80b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.766+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 0x7f44b80b8c90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5611b9e11320 tx=0x7f44c80091f0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.766+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 msgr2=0x7f44d019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.766+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f44c4005940 tx=0x7f44c4004c00 comp rx=0 tx=0).stop 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.766+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 shutdown_connections 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44d0108cf0 0x7f44d01a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44b80b6780 0x7f44b80b8c90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d01083a0 0x7f44d019cce0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 --2- 172.21.15.28:0/2199564172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44d0102a60 0x7f44d019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 >> 172.21.15.28:0/2199564172 conn(0x7f44d0079730 msgr2=0x7f44d01052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 shutdown_connections 2024-09-21T05:52:40.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:40.767+0000 7f44d7320640 1 -- 172.21.15.28:0/2199564172 wait complete. 2024-09-21T05:52:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:40 smithi181 ceph-mon[34819]: pgmap v494: 97 pgs: 97 active+clean; 16 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 32 KiB/s rd, 22 MiB/s wr, 5.05k op/s 2024-09-21T05:52:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:42 smithi129 ceph-mon[34885]: pgmap v495: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 37 KiB/s rd, 32 MiB/s wr, 7.21k op/s 2024-09-21T05:52:42.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:42 smithi028 ceph-mon[31227]: pgmap v495: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 37 KiB/s rd, 32 MiB/s wr, 7.21k op/s 2024-09-21T05:52:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:42 smithi181 ceph-mon[34819]: pgmap v495: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 37 KiB/s rd, 32 MiB/s wr, 7.21k op/s 2024-09-21T05:52:43.094 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:52:43.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:52:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.541+0000 7ff099ce8640 1 -- 172.21.15.28:0/3647061861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094102630 msgr2=0x7ff094106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.541+0000 7ff099ce8640 1 --2- 172.21.15.28:0/3647061861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094102630 0x7ff094106190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff088009860 tx=0x7ff08802a380 comp rx=0 tx=0).stop 2024-09-21T05:52:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.544+0000 7ff099ce8640 1 -- 172.21.15.28:0/3647061861 shutdown_connections 2024-09-21T05:52:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.544+0000 7ff099ce8640 1 --2- 172.21.15.28:0/3647061861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094102630 0x7ff094106190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.544+0000 7ff099ce8640 1 --2- 172.21.15.28:0/3647061861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff094101ce0 0x7ff0941020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.544+0000 7ff099ce8640 1 --2- 172.21.15.28:0/3647061861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff094105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.544+0000 7ff099ce8640 1 -- 172.21.15.28:0/3647061861 >> 172.21.15.28:0/3647061861 conn(0x7ff094077c60 msgr2=0x7ff0940780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.548+0000 7ff099ce8640 1 -- 172.21.15.28:0/3647061861 shutdown_connections 2024-09-21T05:52:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.550+0000 7ff099ce8640 1 -- 172.21.15.28:0/3647061861 wait complete. 2024-09-21T05:52:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.551+0000 7ff099ce8640 1 Processor -- start 2024-09-21T05:52:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.551+0000 7ff099ce8640 1 -- start start 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094101ce0 0x7ff0940743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff094102630 0x7ff094074970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff09410b8a0 con 0x7ff0941053f0 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff09410b720 con 0x7ff094101ce0 2024-09-21T05:52:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.552+0000 7ff099ce8640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff09410ba20 con 0x7ff094102630 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33292/0 (socket says 172.21.15.28:33292) 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 -- 172.21.15.28:0/503507496 learned_addr learned my addr 172.21.15.28:0/503507496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 -- 172.21.15.28:0/503507496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff094102630 msgr2=0x7ff094074970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff094102630 0x7ff094074970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 -- 172.21.15.28:0/503507496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094101ce0 msgr2=0x7ff0940743f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094101ce0 0x7ff0940743f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.554+0000 7ff093fff640 1 -- 172.21.15.28:0/503507496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0941179d0 con 0x7ff0941053f0 2024-09-21T05:52:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.555+0000 7ff093fff640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff08802a8f0 tx=0x7ff088032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.558+0000 7ff090ff9640 1 -- 172.21.15.28:0/503507496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088037050 con 0x7ff0941053f0 2024-09-21T05:52:44.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.558+0000 7ff090ff9640 1 -- 172.21.15.28:0/503507496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff088043070 con 0x7ff0941053f0 2024-09-21T05:52:44.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.558+0000 7ff090ff9640 1 -- 172.21.15.28:0/503507496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088003b20 con 0x7ff0941053f0 2024-09-21T05:52:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.561+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09410fb40 con 0x7ff0941053f0 2024-09-21T05:52:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.561+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09410ffd0 con 0x7ff0941053f0 2024-09-21T05:52:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.567+0000 7ff090ff9640 1 -- 172.21.15.28:0/503507496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff0880327b0 con 0x7ff0941053f0 2024-09-21T05:52:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.569+0000 7ff0a025c740 1 Processor -- start 2024-09-21T05:52:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.569+0000 7ff0a025c740 1 -- start start 2024-09-21T05:52:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.569+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f004862430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.569+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f00449a670 0x55f004862eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0a025c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0047d8e60 con 0x55f00449a670 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0a025c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f0047c8290 con 0x55f0044bb350 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0a025c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f0047cc0a0 con 0x55f004492490 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49252/0 (socket says 172.21.15.28:49252) 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 -- 172.21.15.28:0/759961699 learned_addr learned my addr 172.21.15.28:0/759961699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 -- 172.21.15.28:0/759961699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 msgr2=0x55f004862430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:52:44.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f004862430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 -- 172.21.15.28:0/759961699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f00449a670 msgr2=0x55f004862eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f00449a670 0x55f004862eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.570+0000 7ff0937fe640 1 -- 172.21.15.28:0/759961699 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f004526930 con 0x55f004492490 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.571+0000 7ff0937fe640 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff08401aaa0 tx=0x7ff08401af60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c875f0964c5b0f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.571+0000 7ff090ff9640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 0x7ff07c0b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.572+0000 7ff090ff9640 1 -- 172.21.15.28:0/503507496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff088107720 con 0x7ff0941053f0 2024-09-21T05:52:44.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:44 smithi028 ceph-mon[31227]: pgmap v496: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 7.43k op/s 2024-09-21T05:52:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.574+0000 7ff092ffd640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 0x7ff07c0b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.574+0000 7ff079ffb640 1 -- 172.21.15.28:0/759961699 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084013030 con 0x55f004492490 2024-09-21T05:52:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.574+0000 7ff079ffb640 1 -- 172.21.15.28:0/759961699 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff084005020 con 0x55f004492490 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 -- 172.21.15.28:0/759961699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 msgr2=0x55f004862970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff08401aaa0 tx=0x7ff08401af60 comp rx=0 tx=0).stop 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 -- 172.21.15.28:0/759961699 shutdown_connections 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f00449a670 0x55f004862eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f004492490 0x55f004862970 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 --2- 172.21.15.28:0/759961699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f004862430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.576+0000 7ff0a025c740 1 -- 172.21.15.28:0/759961699 >> 172.21.15.28:0/759961699 conn(0x55f0044af740 msgr2=0x55f004860320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.577+0000 7ff0a025c740 1 -- 172.21.15.28:0/759961699 shutdown_connections 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.577+0000 7ff0a025c740 1 -- 172.21.15.28:0/759961699 wait complete. 2024-09-21T05:52:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.577+0000 7ff0a025c740 1 -- start start 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.577+0000 7ff0a025c740 1 Processor -- start 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f00486df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00487a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f004629890 con 0x55f004492490 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f004762d40 con 0x55f0044bb350 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f004870c60 con 0x55f00449a670 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff0a025c740 10 client.0 _ll_register_callbacks cb 0x55f0047c6c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff092ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff092ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49266/0 (socket says 172.21.15.28:49266) 2024-09-21T05:52:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.578+0000 7ff092ffd640 1 -- 172.21.15.28:0/1970464897 learned_addr learned my addr 172.21.15.28:0/1970464897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.579+0000 7ff093fff640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00487a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff0937fe640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f00486df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff093fff640 1 -- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 msgr2=0x55f00486e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff093fff640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff093fff640 1 -- 172.21.15.28:0/1970464897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 msgr2=0x55f00486df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff093fff640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f00486df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.580+0000 7ff093fff640 1 -- 172.21.15.28:0/1970464897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00487a570 con 0x55f0044bb350 2024-09-21T05:52:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.584+0000 7ff092ffd640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 0x7ff07c0b8d90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00aee0 tx=0x7ff08c010800 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.585+0000 7ff093fff640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00487a030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff088104e30 tx=0x7ff0880cce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35873871ab23f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.587+0000 7ff072ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:52:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.587+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0881052c0 con 0x55f0044bb350 2024-09-21T05:52:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.587+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff0880d6240 con 0x55f0044bb350 2024-09-21T05:52:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.590+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff084005020 con 0x55f0044bb350 2024-09-21T05:52:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.591+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff0880cd150 con 0x55f0044bb350 2024-09-21T05:52:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.591+0000 7ff0a025c740 10 client.26504 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.591+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f00487cea0 con 0x55f0044bb350 2024-09-21T05:52:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.592+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff084005020 con 0x55f0044bb350 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.592+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7ff0880c8f90 con 0x55f0044bb350 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.592+0000 7ff0a025c740 10 client.26504 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.592+0000 7ff0a025c740 10 client.26504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.592+0000 7ff0a025c740 20 client.26504 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.593+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f004479cd0 0x55f0048805c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.593+0000 7ff0a025c740 4 client.26504 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.593+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0047dadb0 con 0x55f004479cd0 2024-09-21T05:52:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.596+0000 7ff0937fe640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f004479cd0 0x55f0048805c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.602+0000 7ff0937fe640 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f004479cd0 0x55f0048805c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f167c9ade44ad87 server_cookie=7dcaa5eb0f7e488 in_seq=0 out_seq=0 2024-09-21T05:52:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.602+0000 7ff072ffd640 10 client.26504 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.606+0000 7ff072ffd640 1 -- 172.21.15.28:0/1970464897 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f0047dadb0 con 0x55f004479cd0 2024-09-21T05:52:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.606+0000 7ff072ffd640 10 client.26504 handle_command_reply: tid=0 2024-09-21T05:52:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.606+0000 7ff0a025c740 1 client.26504 shutdown 2024-09-21T05:52:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.607+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 msgr2=0x55f00487a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.607+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00487a030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff088104e30 tx=0x7ff0880cce70 comp rx=0 tx=0).stop 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 shutdown_connections 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f004479cd0 0x55f0048805c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00487a030 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 --2- 172.21.15.28:0/1970464897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f00486df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 >> 172.21.15.28:0/1970464897 conn(0x55f0044af740 msgr2=0x55f0048655c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff072ffd640 0 client.26504 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 shutdown_connections 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.611+0000 7ff0a025c740 1 -- 172.21.15.28:0/1970464897 wait complete. 2024-09-21T05:52:44.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.612+0000 7ff0a025c740 20 client.26504 trim_cache size 0 max 16384 2024-09-21T05:52:44.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:44 smithi129 ceph-mon[34885]: pgmap v496: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 7.43k op/s 2024-09-21T05:52:44.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.649+0000 7ff0a025c740 1 Processor -- start 2024-09-21T05:52:44.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.649+0000 7ff0a025c740 1 -- start start 2024-09-21T05:52:44.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f004879f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00486b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000e8c0 con 0x55f004492490 2024-09-21T05:52:44.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff080010430 con 0x55f0044bb350 2024-09-21T05:52:44.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.650+0000 7ff0a025c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000ef90 con 0x55f00449a670 2024-09-21T05:52:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.651+0000 7ff093fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00486b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.651+0000 7ff092ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.651+0000 7ff092ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49274/0 (socket says 172.21.15.28:49274) 2024-09-21T05:52:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.651+0000 7ff092ffd640 1 -- 172.21.15.28:0/3899045925 learned_addr learned my addr 172.21.15.28:0/3899045925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.656+0000 7ff092ffd640 1 -- 172.21.15.28:0/3899045925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 msgr2=0x55f00486b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.656+0000 7ff092ffd640 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00486b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.656+0000 7ff092ffd640 1 -- 172.21.15.28:0/3899045925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 msgr2=0x55f004879f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.656+0000 7ff092ffd640 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f004879f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.656+0000 7ff092ffd640 1 -- 172.21.15.28:0/3899045925 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f004864b60 con 0x55f00449a670 2024-09-21T05:52:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.663+0000 7ff092ffd640 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0102c0 tx=0x7ff08c009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfcf6433323c7564 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.667+0000 7ff0797fa640 1 -- 172.21.15.28:0/3899045925 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c020070 con 0x55f00449a670 2024-09-21T05:52:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.667+0000 7ff0797fa640 1 -- 172.21.15.28:0/3899045925 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c025910 con 0x55f00449a670 2024-09-21T05:52:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.667+0000 7ff0797fa640 1 -- 172.21.15.28:0/3899045925 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00d930 con 0x55f00449a670 2024-09-21T05:52:44.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.669+0000 7ff0a025c740 1 -- 172.21.15.28:0/3899045925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 msgr2=0x55f00486af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.669+0000 7ff0a025c740 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0102c0 tx=0x7ff08c009230 comp rx=0 tx=0).stop 2024-09-21T05:52:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.672+0000 7ff0a025c740 1 -- 172.21.15.28:0/3899045925 shutdown_connections 2024-09-21T05:52:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.672+0000 7ff0a025c740 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f0044bb350 0x55f00486b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.672+0000 7ff0a025c740 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f00486af60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.672+0000 7ff0a025c740 1 --2- 172.21.15.28:0/3899045925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f004492490 0x55f004879f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.672+0000 7ff0a025c740 1 -- 172.21.15.28:0/3899045925 >> 172.21.15.28:0/3899045925 conn(0x55f0044af740 msgr2=0x55f00486c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.677+0000 7ff0a025c740 1 -- 172.21.15.28:0/3899045925 shutdown_connections 2024-09-21T05:52:44.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.677+0000 7ff0a025c740 1 -- 172.21.15.28:0/3899045925 wait complete. 2024-09-21T05:52:44.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.677+0000 7ff0a025c740 1 -- start start 2024-09-21T05:52:44.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.678+0000 7ff0a025c740 1 Processor -- start 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.679+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f004492490 0x55f0048726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.679+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f004872c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.680+0000 7ff0a025c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.680+0000 7ff0a025c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000fd60 con 0x55f0044bb350 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.680+0000 7ff0a025c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000f660 con 0x55f004492490 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.680+0000 7ff0a025c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000c8a0 con 0x55f00449a670 2024-09-21T05:52:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.680+0000 7ff0a025c740 10 client.0 _ll_register_callbacks cb 0x7ff088005e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:44.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.681+0000 7ff092ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f004872c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.681+0000 7ff093fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.681+0000 7ff093fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33330/0 (socket says 172.21.15.28:33330) 2024-09-21T05:52:44.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.681+0000 7ff093fff640 1 -- 172.21.15.28:0/2889637890 learned_addr learned my addr 172.21.15.28:0/2889637890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:44.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.681+0000 7ff0937fe640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f004492490 0x55f0048726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.684+0000 7ff093fff640 1 -- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 msgr2=0x55f004872c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.684+0000 7ff093fff640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f004872c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.684+0000 7ff093fff640 1 -- 172.21.15.28:0/2889637890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f004492490 msgr2=0x55f0048726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.684+0000 7ff093fff640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f004492490 0x55f0048726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.684+0000 7ff093fff640 1 -- 172.21.15.28:0/2889637890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0048ea510 con 0x55f0044bb350 2024-09-21T05:52:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.685+0000 7ff093fff640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cd2f0 tx=0x7ff0880cd000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e9dec4389c5d30 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:44.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.687+0000 7ff072ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:52:44.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.687+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0880ec030 con 0x55f0044bb350 2024-09-21T05:52:44.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.687+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff08c025910 con 0x55f0044bb350 2024-09-21T05:52:44.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.687+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7ff0880d6040 con 0x55f0044bb350 2024-09-21T05:52:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.689+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff0880e7050 con 0x55f0044bb350 2024-09-21T05:52:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.689+0000 7ff0a025c740 10 client.16923 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.689+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f0048eb7d0 con 0x55f0044bb350 2024-09-21T05:52:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.694+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c025910 con 0x55f0044bb350 2024-09-21T05:52:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.695+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7ff0880cc310 con 0x55f0044bb350 2024-09-21T05:52:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.696+0000 7ff0a025c740 10 client.16923 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.696+0000 7ff0a025c740 10 client.16923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.696+0000 7ff0a025c740 20 client.16923 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.697+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f00444f080 0x55f0048eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.697+0000 7ff0a025c740 4 client.16923 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:52:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.697+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f0047d6860 con 0x55f00444f080 2024-09-21T05:52:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.701+0000 7ff0937fe640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f00444f080 0x55f0048eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:44.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.706+0000 7ff0937fe640 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f00444f080 0x55f0048eddd0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed2c8e3e5bc77d4 server_cookie=1c5e0666d074ed86 in_seq=0 out_seq=0 2024-09-21T05:52:44.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.706+0000 7ff072ffd640 10 client.16923 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.710+0000 7ff072ffd640 1 -- 172.21.15.28:0/2889637890 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f0047d6860 con 0x55f00444f080 2024-09-21T05:52:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.710+0000 7ff072ffd640 10 client.16923 handle_command_reply: tid=0 2024-09-21T05:52:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.710+0000 7ff0a025c740 1 client.16923 shutdown 2024-09-21T05:52:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.714+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 msgr2=0x55f0048e9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.714+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cd2f0 tx=0x7ff0880cd000 comp rx=0 tx=0).stop 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 shutdown_connections 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f00444f080 0x55f0048eddd0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0044bb350 0x55f0048e9f80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f00449a670 0x55f004872c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 --2- 172.21.15.28:0/2889637890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f004492490 0x55f0048726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 >> 172.21.15.28:0/2889637890 conn(0x55f0044af740 msgr2=0x55f004863410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff072ffd640 0 client.16923 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 shutdown_connections 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 1 -- 172.21.15.28:0/2889637890 wait complete. 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.718+0000 7ff0a025c740 20 client.16923 trim_cache size 0 max 16384 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:52:44.721 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c989f3e2-1085-4489-9f7a-2cd347e5ea3a", 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 msgr2=0x7ff07c0b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 0x7ff07c0b8d90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00aee0 tx=0x7ff08c010800 comp rx=0 tx=0).stop 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 msgr2=0x7ff09410f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff08802a8f0 tx=0x7ff088032cd0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 shutdown_connections 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff07c0b6880 0x7ff07c0b8d90 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00aee0 tx=0x7ff08c010800 comp rx=0 tx=0).stop 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0941053f0 0x7ff09410f5a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff094102630 0x7ff094074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 --2- 172.21.15.28:0/503507496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff094101ce0 0x7ff0940743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.719+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 >> 172.21.15.28:0/503507496 conn(0x7ff094077c60 msgr2=0x7ff094100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.720+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 shutdown_connections 2024-09-21T05:52:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:44.720+0000 7ff099ce8640 1 -- 172.21.15.28:0/503507496 wait complete. 2024-09-21T05:52:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:44 smithi181 ceph-mon[34819]: pgmap v496: 97 pgs: 97 active+clean; 16 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 47 KiB/s rd, 33 MiB/s wr, 7.43k op/s 2024-09-21T05:52:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:45 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:52:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:45 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:45 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:52:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:45 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:52:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:45 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:52:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:45 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:52:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:45 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:45 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:52:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:45 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:52:45.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:45 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:52:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:45 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:52:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:45 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:45 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:52:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:45 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:52:45.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:45 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:52:45.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:52:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:52:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:46 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:52:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:46 smithi129 ceph-mon[34885]: pgmap v497: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.89k op/s 2024-09-21T05:52:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:46 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:52:46.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:46 smithi028 ceph-mon[31227]: pgmap v497: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.89k op/s 2024-09-21T05:52:46.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:46 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:52:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:46 smithi181 ceph-mon[34819]: pgmap v497: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.89k op/s 2024-09-21T05:52:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.211+0000 7fd5a3290640 1 -- 172.21.15.28:0/2605056197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c11cba0 msgr2=0x7fd59c11efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.211+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2605056197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c11cba0 0x7fd59c11efc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd594009be0 tx=0x7fd59402a770 comp rx=0 tx=0).stop 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.214+0000 7fd5a3290640 1 -- 172.21.15.28:0/2605056197 shutdown_connections 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.214+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2605056197 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c11cba0 0x7fd59c11efc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.214+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2605056197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c10a900 0x7fd59c10ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.214+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2605056197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.214+0000 7fd5a3290640 1 -- 172.21.15.28:0/2605056197 >> 172.21.15.28:0/2605056197 conn(0x7fd59c06ee50 msgr2=0x7fd59c06f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.218+0000 7fd5a3290640 1 -- 172.21.15.28:0/2605056197 shutdown_connections 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.218+0000 7fd5a3290640 1 -- 172.21.15.28:0/2605056197 wait complete. 2024-09-21T05:52:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 Processor -- start 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 -- start start 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 0x7fd59c11c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 0x7fd59c118d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c114ba0 con 0x7fd59c111d70 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c114a20 con 0x7fd59c10a900 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.219+0000 7fd5a3290640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c114d20 con 0x7fd59c11cba0 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.222+0000 7fd5a1a8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.222+0000 7fd5a1a8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33642/0 (socket says 172.21.15.28:33642) 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.222+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/2202266759 learned_addr learned my addr 172.21.15.28:0/2202266759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.222+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 0x7fd59c118d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.222+0000 7fd5a228e640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 0x7fd59c11c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/2202266759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 msgr2=0x7fd59c118d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 0x7fd59c118d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/2202266759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 msgr2=0x7fd59c11c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 0x7fd59c11c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/2202266759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c1195b0 con 0x7fd59c111d70 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 0x7fd59c118d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.226+0000 7fd5a228e640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 0x7fd59c11c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.227+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd58c01ea00 tx=0x7fd58c01eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.227+0000 7fd5937fe640 1 -- 172.21.15.28:0/2202266759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c028070 con 0x7fd59c111d70 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.228+0000 7fd5937fe640 1 -- 172.21.15.28:0/2202266759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd58c021470 con 0x7fd59c111d70 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.228+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59c1acd20 con 0x7fd59c111d70 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.228+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59c1ad1b0 con 0x7fd59c111d70 2024-09-21T05:52:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.228+0000 7fd5937fe640 1 -- 172.21.15.28:0/2202266759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c020630 con 0x7fd59c111d70 2024-09-21T05:52:47.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.232+0000 7fd5937fe640 1 -- 172.21.15.28:0/2202266759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd58c02b070 con 0x7fd59c111d70 2024-09-21T05:52:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.235+0000 7fd5aaa66740 1 Processor -- start 2024-09-21T05:52:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.237+0000 7fd5937fe640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 0x7fd5880b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.237+0000 7fd5937fe640 1 -- 172.21.15.28:0/2202266759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd58c0f8210 con 0x7fd59c111d70 2024-09-21T05:52:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.243+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 0x7fd5880b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 -- start start 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a5c1b0 0x558f24e2cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e2d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f24da2e50 con 0x558f24a64390 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558f24d92280 con 0x558f24a85070 2024-09-21T05:52:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558f24d96090 con 0x558f24a5c1b0 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 0x7fd5880b8d10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd59402ace0 tx=0x7fd59402ac70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5a228e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5a228e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33150/0 (socket says 172.21.15.28:33150) 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.246+0000 7fd5a228e640 1 -- 172.21.15.28:0/1271994840 learned_addr learned my addr 172.21.15.28:0/1271994840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a5c1b0 0x558f24e2cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 -- 172.21.15.28:0/1271994840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a5c1b0 msgr2=0x558f24e2cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a5c1b0 0x558f24e2cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 -- 172.21.15.28:0/1271994840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 msgr2=0x558f24e2d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e2d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 -- 172.21.15.28:0/1271994840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f24af0650 con 0x558f24a85070 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.247+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd59800bcb0 tx=0x7fd59800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca64322fc6e15ff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.248+0000 7fd590ff9640 1 -- 172.21.15.28:0/1271994840 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598015070 con 0x558f24a85070 2024-09-21T05:52:47.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.248+0000 7fd590ff9640 1 -- 172.21.15.28:0/1271994840 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5980035f0 con 0x558f24a85070 2024-09-21T05:52:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1271994840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 msgr2=0x558f24e2c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd59800bcb0 tx=0x7fd59800f730 comp rx=0 tx=0).stop 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1271994840 shutdown_connections 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e2d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a5c1b0 0x558f24e2cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1271994840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a85070 0x558f24e2c810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.249+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1271994840 >> 172.21.15.28:0/1271994840 conn(0x558f24a79460 msgr2=0x558f24e2a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.250+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1271994840 shutdown_connections 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.250+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1271994840 wait complete. 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.250+0000 7fd5aaa66740 1 -- start start 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 Processor -- start 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e3a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 0x558f24e37a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 0x558f24e38130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f24bf3700 con 0x558f24a439f0 2024-09-21T05:52:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558f24d2d110 con 0x558f24a5c1b0 2024-09-21T05:52:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558f24e3b040 con 0x558f24a64390 2024-09-21T05:52:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.251+0000 7fd5aaa66740 10 client.0 _ll_register_callbacks cb 0x558f24d46990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.254+0000 7fd5a2a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 0x558f24e38130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.254+0000 7fd5a2a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 0x558f24e38130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33670/0 (socket says 172.21.15.28:33670) 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.254+0000 7fd5a2a8f640 1 -- 172.21.15.28:0/1281964410 learned_addr learned my addr 172.21.15.28:0/1281964410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.255+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 0x558f24e37a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.255+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e3a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.256+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/1281964410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 msgr2=0x558f24e3a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.256+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e3a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.257+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/1281964410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 msgr2=0x558f24e38130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.257+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 0x558f24e38130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.257+0000 7fd5a1a8d640 1 -- 172.21.15.28:0/1281964410 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f24d5ac40 con 0x558f24a64390 2024-09-21T05:52:47.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.264+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 0x558f24e37a40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0f5980 tx=0x7fd58c0bd8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5344ef52d8660301 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.264+0000 7fd585ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:47.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.264+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c032040 con 0x558f24a64390 2024-09-21T05:52:47.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.265+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558f24da4da0 con 0x558f24a64390 2024-09-21T05:52:47.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.265+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd58c0a93b0 con 0x558f24a64390 2024-09-21T05:52:47.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.266+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fd58c0c2b60 con 0x558f24a64390 2024-09-21T05:52:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.268+0000 7fd5aaa66740 10 client.26527 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.268+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558f24e46a00 con 0x558f24a64390 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.269+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558f24da4da0 con 0x558f24a64390 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fd58c01f040 con 0x558f24a64390 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 10 client.26527 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 20 client.26527 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24a18da0 0x558f24e4a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 4 client.26527 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f24da0850 con 0x558f24a18da0 2024-09-21T05:52:47.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.270+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24a18da0 0x558f24e4a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.272+0000 7fd5a228e640 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24a18da0 0x558f24e4a370 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdb2a491bfde1e92 server_cookie=716cb14168217e06 in_seq=0 out_seq=0 2024-09-21T05:52:47.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.272+0000 7fd585ffb640 10 client.26527 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:47.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.273+0000 7fd585ffb640 1 -- 172.21.15.28:0/1281964410 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558f24da0850 con 0x558f24a18da0 2024-09-21T05:52:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.273+0000 7fd585ffb640 10 client.26527 handle_command_reply: tid=0 2024-09-21T05:52:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.274+0000 7fd5aaa66740 1 client.26527 shutdown 2024-09-21T05:52:47.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.278+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 msgr2=0x558f24e37a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.278+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 0x558f24e37a40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0f5980 tx=0x7fd58c0bd8b0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 shutdown_connections 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24a18da0 0x558f24e4a370 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a439f0 0x558f24e38130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a64390 0x558f24e37a40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1281964410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e3a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 >> 172.21.15.28:0/1281964410 conn(0x558f24a79460 msgr2=0x558f24e2f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.279+0000 7fd585ffb640 0 client.26527 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.285+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 shutdown_connections 2024-09-21T05:52:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.286+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1281964410 wait complete. 2024-09-21T05:52:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.286+0000 7fd5aaa66740 20 client.26527 trim_cache size 0 max 16384 2024-09-21T05:52:47.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.321+0000 7fd5aaa66740 1 Processor -- start 2024-09-21T05:52:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.322+0000 7fd5aaa66740 1 -- start start 2024-09-21T05:52:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.322+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.322+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.322+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b5c50 con 0x558f24a64390 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b77c0 con 0x558f24a5c1b0 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5aaa66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b6320 con 0x558f24a439f0 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5a2a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5a2a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33672/0 (socket says 172.21.15.28:33672) 2024-09-21T05:52:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.323+0000 7fd5a2a8f640 1 -- 172.21.15.28:0/1656522558 learned_addr learned my addr 172.21.15.28:0/1656522558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:47.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.324+0000 7fd5a2a8f640 1 -- 172.21.15.28:0/1656522558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 msgr2=0x558f24e331f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:52:47.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.324+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e331f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.324+0000 7fd5a2a8f640 1 -- 172.21.15.28:0/1656522558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 msgr2=0x558f24e256a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.324+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.324+0000 7fd5a2a8f640 1 -- 172.21.15.28:0/1656522558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f24e35f40 con 0x558f24a64390 2024-09-21T05:52:47.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.325+0000 7fd5a2a8f640 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd594034040 tx=0x7fd59402a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42dfb4c1ce2a8fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.325+0000 7fd587fff640 1 -- 172.21.15.28:0/1656522558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59404d070 con 0x558f24a64390 2024-09-21T05:52:47.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.325+0000 7fd587fff640 1 -- 172.21.15.28:0/1656522558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd594042030 con 0x558f24a64390 2024-09-21T05:52:47.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.325+0000 7fd587fff640 1 -- 172.21.15.28:0/1656522558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594036040 con 0x558f24a64390 2024-09-21T05:52:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1656522558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 msgr2=0x558f24e32030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd594034040 tx=0x7fd59402a770 comp rx=0 tx=0).stop 2024-09-21T05:52:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1656522558 shutdown_connections 2024-09-21T05:52:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a64390 0x558f24e32030 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a5c1b0 0x558f24e256a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/1656522558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.327+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1656522558 >> 172.21.15.28:0/1656522558 conn(0x558f24a79460 msgr2=0x558f24e31920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.329+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1656522558 shutdown_connections 2024-09-21T05:52:47.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.329+0000 7fd5aaa66740 1 -- 172.21.15.28:0/1656522558 wait complete. 2024-09-21T05:52:47.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.330+0000 7fd5aaa66740 1 -- start start 2024-09-21T05:52:47.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.330+0000 7fd5aaa66740 1 Processor -- start 2024-09-21T05:52:47.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.331+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.331+0000 7fd5aaa66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a5c1b0 0x558f24e3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.331+0000 7fd5a228e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.331+0000 7fd5a228e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:49396/0 (socket says 172.21.15.28:49396) 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.331+0000 7fd5a228e640 1 -- 172.21.15.28:0/3398298625 learned_addr learned my addr 172.21.15.28:0/3398298625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.332+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a64390 0x558f24eb3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.332+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b70f0 con 0x558f24a5c1b0 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.332+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b69f0 con 0x558f24a64390 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.332+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1b3c30 con 0x558f24a439f0 2024-09-21T05:52:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.332+0000 7fd5aaa66740 10 client.0 _ll_register_callbacks cb 0x7fd58c0ba6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.333+0000 7fd5a228e640 1 -- 172.21.15.28:0/3398298625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a64390 msgr2=0x558f24eb3d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:52:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.333+0000 7fd5a228e640 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a64390 0x558f24eb3d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.333+0000 7fd5a228e640 1 -- 172.21.15.28:0/3398298625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a5c1b0 msgr2=0x558f24e3cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.333+0000 7fd5a228e640 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a5c1b0 0x558f24e3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.333+0000 7fd5a228e640 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f24eb42b0 con 0x558f24a439f0 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.334+0000 7fd5a228e640 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd59800bc80 tx=0x7fd59800ba00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1323fe94f3d66e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.334+0000 7fd585ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.334+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598015040 con 0x558f24a439f0 2024-09-21T05:52:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.334+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd594042030 con 0x558f24a439f0 2024-09-21T05:52:47.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.334+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd5980042d0 con 0x558f24a439f0 2024-09-21T05:52:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.342+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fd598005040 con 0x558f24a439f0 2024-09-21T05:52:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.344+0000 7fd5aaa66740 10 client.26530 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.344+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558f24eb5490 con 0x558f24a439f0 2024-09-21T05:52:47.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.347+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd594042030 con 0x558f24a439f0 2024-09-21T05:52:47.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.347+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fd598032070 con 0x558f24a439f0 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 10 client.26530 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 10 client.26530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 20 client.26530 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24eb5a30 0x558f24eb7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 4 client.26530 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.349+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558f24d9dc80 con 0x558f24eb5a30 2024-09-21T05:52:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.350+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24eb5a30 0x558f24eb7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd5a1a8d640 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24eb5a30 0x558f24eb7e50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daea2b861bf9d96c server_cookie=b98c3a21e44aff58 in_seq=0 out_seq=0 2024-09-21T05:52:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd585ffb640 10 client.26530 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd585ffb640 1 -- 172.21.15.28:0/3398298625 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2850 (crc 0 0 0) 0x558f24d9dc80 con 0x558f24eb5a30 2024-09-21T05:52:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd585ffb640 10 client.26530 handle_command_reply: tid=0 2024-09-21T05:52:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd5aaa66740 1 client.26530 shutdown 2024-09-21T05:52:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 msgr2=0x558f24e3c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.352+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd59800bc80 tx=0x7fd59800ba00 comp rx=0 tx=0).stop 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 shutdown_connections 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x558f24eb5a30 0x558f24eb7e50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558f24a64390 0x558f24eb3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f24a5c1b0 0x558f24e3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 --2- 172.21.15.28:0/3398298625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558f24a439f0 0x558f24e3c9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 >> 172.21.15.28:0/3398298625 conn(0x558f24a79460 msgr2=0x558f24e2d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd585ffb640 0 client.26530 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 shutdown_connections 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 1 -- 172.21.15.28:0/3398298625 wait complete. 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.358+0000 7fd5aaa66740 20 client.26530 trim_cache size 0 max 16384 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (119 inodes in the stack)", 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:52:47.362 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c989f3e2-1085-4489-9f7a-2cd347e5ea3a", 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:52:47.366 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:52:47.367 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:52:47.368 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 msgr2=0x7fd5880b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 0x7fd5880b8d10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd59402ace0 tx=0x7fd59402ac70 comp rx=0 tx=0).stop 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 msgr2=0x7fd59c118790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd58c01ea00 tx=0x7fd58c01eec0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 shutdown_connections 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.359+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd59c11cba0 0x7fd59c118d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd5880b6800 0x7fd5880b8d10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd59c111d70 0x7fd59c118790 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 --2- 172.21.15.28:0/2202266759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59c10a900 0x7fd59c11c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 >> 172.21.15.28:0/2202266759 conn(0x7fd59c06ee50 msgr2=0x7fd59c073c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 shutdown_connections 2024-09-21T05:52:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:52:47.360+0000 7fd5a3290640 1 -- 172.21.15.28:0/2202266759 wait complete. 2024-09-21T05:52:48.305 INFO:tasks.cephfs.filesystem:scrub status for tag:c989f3e2-1085-4489-9f7a-2cd347e5ea3a - {'path': '/', 'tag': 'c989f3e2-1085-4489-9f7a-2cd347e5ea3a', 'options': 'recursive,force'} 2024-09-21T05:52:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:48 smithi028 ceph-mon[31227]: pgmap v498: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.84k op/s 2024-09-21T05:52:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:48 smithi129 ceph-mon[34885]: pgmap v498: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.84k op/s 2024-09-21T05:52:48.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:48 smithi181 ceph-mon[34819]: pgmap v498: 97 pgs: 97 active+clean; 17 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 41 KiB/s rd, 41 MiB/s wr, 8.84k op/s 2024-09-21T05:52:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.939+0000 7fa7532a2640 1 -- 172.21.15.28:0/2084094838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c1143c0 msgr2=0x7fa74c116890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.939+0000 7fa7532a2640 1 --2- 172.21.15.28:0/2084094838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c1143c0 0x7fa74c116890 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009900 tx=0x7fa73c02a420 comp rx=0 tx=0).stop 2024-09-21T05:52:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.945+0000 7fa7532a2640 1 -- 172.21.15.28:0/2084094838 shutdown_connections 2024-09-21T05:52:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.945+0000 7fa7532a2640 1 --2- 172.21.15.28:0/2084094838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c1143c0 0x7fa74c116890 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.945+0000 7fa7532a2640 1 --2- 172.21.15.28:0/2084094838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 0x7fa74c0761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.945+0000 7fa7532a2640 1 --2- 172.21.15.28:0/2084094838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c077390 0x7fa74c0757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.946+0000 7fa7532a2640 1 -- 172.21.15.28:0/2084094838 >> 172.21.15.28:0/2084094838 conn(0x7fa74c0ff500 msgr2=0x7fa74c101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:48.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.954+0000 7fa7532a2640 1 -- 172.21.15.28:0/2084094838 shutdown_connections 2024-09-21T05:52:48.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.959+0000 7fa7532a2640 1 -- 172.21.15.28:0/2084094838 wait complete. 2024-09-21T05:52:48.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 Processor -- start 2024-09-21T05:52:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 -- start start 2024-09-21T05:52:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 0x7fa74c074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c077390 0x7fa74c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c119fd0 con 0x7fa74c077390 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c119e50 con 0x7fa74c1143c0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.960+0000 7fa7532a2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c11a150 con 0x7fa74c075dd0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75103e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 0x7fa74c074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33194/0 (socket says 172.21.15.28:33194) 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 -- 172.21.15.28:0/467970501 learned_addr learned my addr 172.21.15.28:0/467970501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75083d640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c077390 0x7fa74c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 -- 172.21.15.28:0/467970501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 msgr2=0x7fa74c074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 0x7fa74c074e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 -- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c077390 msgr2=0x7fa74c06e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c077390 0x7fa74c06e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.961+0000 7fa75183f640 1 -- 172.21.15.28:0/467970501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c06ef30 con 0x7fa74c1143c0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.962+0000 7fa75183f640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa73c02a990 tx=0x7fa73c003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:48.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.963+0000 7fa7427fc640 1 -- 172.21.15.28:0/467970501 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c03f070 con 0x7fa74c1143c0 2024-09-21T05:52:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.963+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74c073b50 con 0x7fa74c1143c0 2024-09-21T05:52:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.963+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74c074040 con 0x7fa74c1143c0 2024-09-21T05:52:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.963+0000 7fa7427fc640 1 -- 172.21.15.28:0/467970501 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa73c03bd20 con 0x7fa74c1143c0 2024-09-21T05:52:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.963+0000 7fa7427fc640 1 -- 172.21.15.28:0/467970501 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c035030 con 0x7fa74c1143c0 2024-09-21T05:52:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.968+0000 7fa75a216740 1 Processor -- start 2024-09-21T05:52:48.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 -- start start 2024-09-21T05:52:48.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d0479b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc792f0 0x55db9d047ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db9cfbc6f0 con 0x55db9cc814d0 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55db9cfabb20 con 0x55db9cca21b0 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa75a216740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55db9cfaf930 con 0x55db9cc792f0 2024-09-21T05:52:48.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.972+0000 7fa7427fc640 1 -- 172.21.15.28:0/467970501 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa73c042070 con 0x7fa74c1143c0 2024-09-21T05:52:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.975+0000 7fa7427fc640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 0x7fa7380b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.976+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.976+0000 7fa7427fc640 1 -- 172.21.15.28:0/467970501 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa73c107e50 con 0x7fa74c1143c0 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.976+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33696/0 (socket says 172.21.15.28:33696) 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.976+0000 7fa75083d640 1 -- 172.21.15.28:0/3766842029 learned_addr learned my addr 172.21.15.28:0/3766842029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.976+0000 7fa75083d640 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d0479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.977+0000 7fa75183f640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 0x7fa7380b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.980+0000 7fa75103e640 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc792f0 0x55db9d047ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75183f640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 0x7fa7380b8d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa73c03bfd0 tx=0x7fa73c009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75083d640 1 -- 172.21.15.28:0/3766842029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc792f0 msgr2=0x55db9d047ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75083d640 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc792f0 0x55db9d047ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75083d640 1 -- 172.21.15.28:0/3766842029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 msgr2=0x55db9d0479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75083d640 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d0479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.983+0000 7fa75083d640 1 -- 172.21.15.28:0/3766842029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9cf71530 con 0x55db9cc814d0 2024-09-21T05:52:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.986+0000 7fa75083d640 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa748033740 tx=0x7fa74800a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f351b9d727bc88de server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.991+0000 7fa7377fe640 1 -- 172.21.15.28:0/3766842029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748004020 con 0x55db9cc814d0 2024-09-21T05:52:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.992+0000 7fa7377fe640 1 -- 172.21.15.28:0/3766842029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7480054d0 con 0x55db9cc814d0 2024-09-21T05:52:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.992+0000 7fa7377fe640 1 -- 172.21.15.28:0/3766842029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748004b20 con 0x55db9cc814d0 2024-09-21T05:52:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.993+0000 7fa75a216740 1 -- 172.21.15.28:0/3766842029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 msgr2=0x55db9d048430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.993+0000 7fa75a216740 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa748033740 tx=0x7fa74800a6a0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.996+0000 7fa75a216740 1 -- 172.21.15.28:0/3766842029 shutdown_connections 2024-09-21T05:52:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.996+0000 7fa75a216740 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d048430 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.996+0000 7fa75a216740 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc792f0 0x55db9d047ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.996+0000 7fa75a216740 1 --2- 172.21.15.28:0/3766842029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d0479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:48.996+0000 7fa75a216740 1 -- 172.21.15.28:0/3766842029 >> 172.21.15.28:0/3766842029 conn(0x55db9cc96300 msgr2=0x55db9d0458a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.003+0000 7fa75a216740 1 -- 172.21.15.28:0/3766842029 shutdown_connections 2024-09-21T05:52:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.007+0000 7fa75a216740 1 -- 172.21.15.28:0/3766842029 wait complete. 2024-09-21T05:52:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.007+0000 7fa75a216740 1 -- start start 2024-09-21T05:52:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.007+0000 7fa75a216740 1 Processor -- start 2024-09-21T05:52:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.008+0000 7fa75a216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 0x55db9d053680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d05f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db9cee3ab0 con 0x55db9cc792f0 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55db9cf51730 con 0x55db9cc814d0 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55db9d0560d0 con 0x55db9cca21b0 2024-09-21T05:52:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.009+0000 7fa75a216740 10 client.0 _ll_register_callbacks cb 0x55db9cfa56b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.011+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.011+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33698/0 (socket says 172.21.15.28:33698) 2024-09-21T05:52:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.011+0000 7fa75083d640 1 -- 172.21.15.28:0/3106776980 learned_addr learned my addr 172.21.15.28:0/3106776980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.013+0000 7fa75103e640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 0x55db9d053680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 -- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 msgr2=0x55db9d05f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d05f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 -- 172.21.15.28:0/3106776980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 msgr2=0x55db9d053140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 -- 172.21.15.28:0/3106776980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9cd0d790 con 0x55db9cc814d0 2024-09-21T05:52:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.014+0000 7fa75103e640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 0x55db9d053680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa744005fd0 tx=0x7fa744004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5a51adf3ea75b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.016+0000 7fa734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:52:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.016+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744012850 con 0x55db9cc814d0 2024-09-21T05:52:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.016+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa7440129f0 con 0x55db9cc814d0 2024-09-21T05:52:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.016+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7480054d0 con 0x55db9cc814d0 2024-09-21T05:52:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.019+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa744012bf0 con 0x55db9cc814d0 2024-09-21T05:52:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.019+0000 7fa75a216740 10 client.26534 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.019+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55db9cd86620 con 0x55db9cc814d0 2024-09-21T05:52:49.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7480054d0 con 0x55db9cc814d0 2024-09-21T05:52:49.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fa74400f040 con 0x55db9cc814d0 2024-09-21T05:52:49.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 10 client.26534 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:49.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 10 client.26534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:49.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 20 client.26534 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:49.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc60d80 0x55db9d0652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 4 client.26534 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:52:49.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db9cfbe640 con 0x55db9cc60d80 2024-09-21T05:52:49.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.021+0000 7fa75083d640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc60d80 0x55db9d0652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.030+0000 7fa75083d640 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc60d80 0x55db9d0652f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ada0a4b3e4ba7752 server_cookie=1c4c3edcb235a190 in_seq=0 out_seq=0 2024-09-21T05:52:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.031+0000 7fa734ff9640 10 client.26534 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.033+0000 7fa734ff9640 1 -- 172.21.15.28:0/3106776980 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55db9cfbe640 con 0x55db9cc60d80 2024-09-21T05:52:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.033+0000 7fa734ff9640 10 client.26534 handle_command_reply: tid=0 2024-09-21T05:52:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.033+0000 7fa75a216740 1 client.26534 shutdown 2024-09-21T05:52:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.034+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 msgr2=0x55db9d053680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.034+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 0x55db9d053680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa744005fd0 tx=0x7fa744004410 comp rx=0 tx=0).stop 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 shutdown_connections 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc60d80 0x55db9d0652f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d05f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc814d0 0x55db9d053680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 --2- 172.21.15.28:0/3106776980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.035+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 >> 172.21.15.28:0/3106776980 conn(0x55db9cc96300 msgr2=0x55db9d0489e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.037+0000 7fa734ff9640 0 client.26534 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.041+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 shutdown_connections 2024-09-21T05:52:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.043+0000 7fa75a216740 1 -- 172.21.15.28:0/3106776980 wait complete. 2024-09-21T05:52:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.047+0000 7fa75a216740 20 client.26534 trim_cache size 0 max 16384 2024-09-21T05:52:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.080+0000 7fa75a216740 1 Processor -- start 2024-09-21T05:52:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.080+0000 7fa75a216740 1 -- start start 2024-09-21T05:52:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.080+0000 7fa75a216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.081+0000 7fa75a216740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc814d0 0x55db9d0490f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.081+0000 7fa75a216740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d05f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.081+0000 7fa75a216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a97a0 con 0x55db9cc792f0 2024-09-21T05:52:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.081+0000 7fa75a216740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a7530 con 0x55db9cca21b0 2024-09-21T05:52:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.081+0000 7fa75a216740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a90a0 con 0x55db9cc814d0 2024-09-21T05:52:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.082+0000 7fa75083d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d05f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.082+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.082+0000 7fa75083d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33706/0 (socket says 172.21.15.28:33706) 2024-09-21T05:52:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.082+0000 7fa75083d640 1 -- 172.21.15.28:0/2968977310 learned_addr learned my addr 172.21.15.28:0/2968977310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.083+0000 7fa75103e640 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc814d0 0x55db9d0490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.088+0000 7fa75083d640 1 -- 172.21.15.28:0/2968977310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc814d0 msgr2=0x55db9d0490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.088+0000 7fa75083d640 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc814d0 0x55db9d0490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.088+0000 7fa75083d640 1 -- 172.21.15.28:0/2968977310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 msgr2=0x55db9d053620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.088+0000 7fa75083d640 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.088+0000 7fa75083d640 1 -- 172.21.15.28:0/2968977310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9d051620 con 0x55db9cca21b0 2024-09-21T05:52:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.089+0000 7fa75083d640 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d05f2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7480058c0 tx=0x7fa7480045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe2b1e89891b8a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.094+0000 7fa736ffd640 1 -- 172.21.15.28:0/2968977310 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748043040 con 0x55db9cca21b0 2024-09-21T05:52:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.094+0000 7fa736ffd640 1 -- 172.21.15.28:0/2968977310 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa748005570 con 0x55db9cca21b0 2024-09-21T05:52:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.095+0000 7fa75a216740 1 -- 172.21.15.28:0/2968977310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 msgr2=0x55db9d05f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.095+0000 7fa75a216740 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d05f2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7480058c0 tx=0x7fa7480045d0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.096+0000 7fa75a216740 1 -- 172.21.15.28:0/2968977310 shutdown_connections 2024-09-21T05:52:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.096+0000 7fa75a216740 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cca21b0 0x55db9d05f2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.096+0000 7fa75a216740 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cc814d0 0x55db9d0490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.096+0000 7fa75a216740 1 --2- 172.21.15.28:0/2968977310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc792f0 0x55db9d053620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.096+0000 7fa75a216740 1 -- 172.21.15.28:0/2968977310 >> 172.21.15.28:0/2968977310 conn(0x55db9cc96300 msgr2=0x55db9d0489e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.100+0000 7fa75a216740 1 -- 172.21.15.28:0/2968977310 shutdown_connections 2024-09-21T05:52:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.103+0000 7fa75a216740 1 -- 172.21.15.28:0/2968977310 wait complete. 2024-09-21T05:52:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.104+0000 7fa75a216740 1 -- start start 2024-09-21T05:52:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.104+0000 7fa75a216740 1 Processor -- start 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d056c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d0cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a82d0 con 0x55db9cc814d0 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a7c00 con 0x55db9cc792f0 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1a89d0 con 0x55db9cca21b0 2024-09-21T05:52:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75a216740 10 client.0 _ll_register_callbacks cb 0x7fa744009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:52:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.105+0000 7fa75083d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:33232/0 (socket says 172.21.15.28:33232) 2024-09-21T05:52:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 -- 172.21.15.28:0/1241703423 learned_addr learned my addr 172.21.15.28:0/1241703423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:52:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d0cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75103e640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d056c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 -- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 msgr2=0x55db9d0cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d0cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 -- 172.21.15.28:0/1241703423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 msgr2=0x55db9d056c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d056c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.106+0000 7fa75083d640 1 -- 172.21.15.28:0/1241703423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9d057510 con 0x55db9cc792f0 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.107+0000 7fa75083d640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa748005e10 tx=0x7fa74800ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7f3d4b772593404 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.107+0000 7fa734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.107+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748043070 con 0x55db9cc792f0 2024-09-21T05:52:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.107+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa748005570 con 0x55db9cc792f0 2024-09-21T05:52:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.107+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa748033050 con 0x55db9cc792f0 2024-09-21T05:52:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.108+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fa748033250 con 0x55db9cc792f0 2024-09-21T05:52:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.108+0000 7fa75a216740 10 client.26540 fetch_fsmap learned FSMap version 41 2024-09-21T05:52:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.108+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55db9d0e02a0 con 0x55db9cc792f0 2024-09-21T05:52:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.109+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa748005570 con 0x55db9cc792f0 2024-09-21T05:52:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.109+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7fa748067020 con 0x55db9cc792f0 2024-09-21T05:52:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 10 client.26540 fetch_fsmap finished waiting for FSMap version 41 2024-09-21T05:52:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 10 client.26540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:52:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 20 client.26540 populate_metadata read hostname 'smithi028' 2024-09-21T05:52:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc361c0 0x55db9d0e2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:52:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 4 client.26540 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:52:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.110+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db9cfba0f0 con 0x55db9cc361c0 2024-09-21T05:52:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.116+0000 7fa75103e640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc361c0 0x55db9d0e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:52:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.122+0000 7fa75103e640 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc361c0 0x55db9d0e2690 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae81ed62ee3285b server_cookie=4c96661d99bbf22f in_seq=0 out_seq=0 2024-09-21T05:52:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.122+0000 7fa734ff9640 10 client.26540 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.849+0000 7fa734ff9640 1 -- 172.21.15.28:0/1241703423 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db9cfba0f0 con 0x55db9cc361c0 2024-09-21T05:52:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.849+0000 7fa734ff9640 10 client.26540 handle_command_reply: tid=0 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.849+0000 7fa75a216740 1 client.26540 shutdown 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.849+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 msgr2=0x55db9d056750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.849+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa748005e10 tx=0x7fa74800ac10 comp rx=0 tx=0).stop 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 shutdown_connections 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55db9cc361c0 0x55db9d0e2690 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db9cca21b0 0x55db9d0cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db9cc814d0 0x55db9d056c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 --2- 172.21.15.28:0/1241703423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db9cc792f0 0x55db9d056750 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 >> 172.21.15.28:0/1241703423 conn(0x55db9cc96300 msgr2=0x55db9d045880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa734ff9640 0 client.26540 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:52:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 shutdown_connections 2024-09-21T05:52:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 1 -- 172.21.15.28:0/1241703423 wait complete. 2024-09-21T05:52:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.850+0000 7fa75a216740 20 client.26540 trim_cache size 0 max 16384 2024-09-21T05:52:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:52:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:52:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:52:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 msgr2=0x7fa7380b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 0x7fa7380b8d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa73c03bfd0 tx=0x7fa73c009260 comp rx=0 tx=0).stop 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 msgr2=0x7fa74c06e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa73c02a990 tx=0x7fa73c003cd0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 shutdown_connections 2024-09-21T05:52:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa74c1143c0 0x7fa74c06e710 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.851+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa7380b6880 0x7fa7380b8d90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.852+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa74c077390 0x7fa74c06e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.852+0000 7fa7532a2640 1 --2- 172.21.15.28:0/467970501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa74c075dd0 0x7fa74c074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.852+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 >> 172.21.15.28:0/467970501 conn(0x7fa74c0ff500 msgr2=0x7fa74c1102c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.852+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 shutdown_connections 2024-09-21T05:52:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:52:49.852+0000 7fa7532a2640 1 -- 172.21.15.28:0/467970501 wait complete. 2024-09-21T05:52:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:50 smithi129 ceph-mon[34885]: pgmap v499: 97 pgs: 97 active+clean; 17 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 46 KiB/s rd, 48 MiB/s wr, 10.47k op/s 2024-09-21T05:52:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:50 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:52:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:50 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:52:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:50 smithi181 ceph-mon[34819]: pgmap v499: 97 pgs: 97 active+clean; 17 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 46 KiB/s rd, 48 MiB/s wr, 10.47k op/s 2024-09-21T05:52:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:50 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:52:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:50 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:52:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:50 smithi028 ceph-mon[31227]: pgmap v499: 97 pgs: 97 active+clean; 17 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 46 KiB/s rd, 48 MiB/s wr, 10.47k op/s 2024-09-21T05:52:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:50 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:52:50.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:50 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:52:52.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:52 smithi028 ceph-mon[31227]: pgmap v500: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 46 KiB/s rd, 62 MiB/s wr, 13.40k op/s 2024-09-21T05:52:52.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:52 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:52:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:52 smithi181 ceph-mon[34819]: pgmap v500: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 46 KiB/s rd, 62 MiB/s wr, 13.40k op/s 2024-09-21T05:52:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:52 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:52:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:52 smithi129 ceph-mon[34885]: pgmap v500: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 46 KiB/s rd, 62 MiB/s wr, 13.40k op/s 2024-09-21T05:52:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:52 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:52:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:52:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:52:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:52:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:53 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:53 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:53.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:52:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:52:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:52:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:53 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:53 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:53.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:52:53.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:52:53.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:52:53.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:53 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:53.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:53 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:54 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:54.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:54 smithi129 ceph-mon[34885]: pgmap v501: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 45 KiB/s rd, 53 MiB/s wr, 11.71k op/s 2024-09-21T05:52:55.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:54 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:55.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:54 smithi181 ceph-mon[34819]: pgmap v501: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 45 KiB/s rd, 53 MiB/s wr, 11.71k op/s 2024-09-21T05:52:55.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:54 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:55.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:54 smithi028 ceph-mon[31227]: pgmap v501: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 45 KiB/s rd, 53 MiB/s wr, 11.71k op/s 2024-09-21T05:52:55.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:55.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:56.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:56.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:56.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:56.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:52:56.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:56 smithi129 ceph-mon[34885]: pgmap v502: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 57 KiB/s rd, 60 MiB/s wr, 13.14k op/s 2024-09-21T05:52:57.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:56 smithi181 ceph-mon[34819]: pgmap v502: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 57 KiB/s rd, 60 MiB/s wr, 13.14k op/s 2024-09-21T05:52:57.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:56 smithi028 ceph-mon[31227]: pgmap v502: 97 pgs: 97 active+clean; 17 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 57 KiB/s rd, 60 MiB/s wr, 13.14k op/s 2024-09-21T05:52:58.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:57 smithi181 ceph-mon[34819]: pgmap v503: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 66 KiB/s rd, 63 MiB/s wr, 13.91k op/s 2024-09-21T05:52:58.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:57 smithi028 ceph-mon[31227]: pgmap v503: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 66 KiB/s rd, 63 MiB/s wr, 13.91k op/s 2024-09-21T05:52:58.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:57 smithi129 ceph-mon[34885]: pgmap v503: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 66 KiB/s rd, 63 MiB/s wr, 13.91k op/s 2024-09-21T05:53:00.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:52:59 smithi181 ceph-mon[34819]: pgmap v504: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 94 KiB/s rd, 57 MiB/s wr, 12.60k op/s 2024-09-21T05:53:00.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:52:59 smithi028 ceph-mon[31227]: pgmap v504: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 94 KiB/s rd, 57 MiB/s wr, 12.60k op/s 2024-09-21T05:53:00.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:52:59 smithi129 ceph-mon[34885]: pgmap v504: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 94 KiB/s rd, 57 MiB/s wr, 12.60k op/s 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: pgmap v505: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 97 KiB/s rd, 52 MiB/s wr, 11.64k op/s 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:02.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:01 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: pgmap v505: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 97 KiB/s rd, 52 MiB/s wr, 11.64k op/s 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:01 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: pgmap v505: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 97 KiB/s rd, 52 MiB/s wr, 11.64k op/s 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:02.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:01 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 -- 172.21.15.28:0/2210297082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 msgr2=0x7f32301086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 --2- 172.21.15.28:0/2210297082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301086f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f322000fb10 tx=0x7f322002e570 comp rx=0 tx=0).stop 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 -- 172.21.15.28:0/2210297082 shutdown_connections 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 --2- 172.21.15.28:0/2210297082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3230108c30 0x7f323010f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 --2- 172.21.15.28:0/2210297082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301086f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 --2- 172.21.15.28:0/2210297082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32301029b0 0x7f3230102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 -- 172.21.15.28:0/2210297082 >> 172.21.15.28:0/2210297082 conn(0x7f323007a740 msgr2=0x7f323007ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 -- 172.21.15.28:0/2210297082 shutdown_connections 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.008+0000 7f3235d20640 1 -- 172.21.15.28:0/2210297082 wait complete. 2024-09-21T05:53:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.009+0000 7f3235d20640 1 Processor -- start 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 -- start start 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32301029b0 0x7f32301a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3230111810 con 0x7f3230108c30 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3230111690 con 0x7f32301029b0 2024-09-21T05:53:03.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.010+0000 7f3235d20640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3230111990 con 0x7f32301082e0 2024-09-21T05:53:03.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49968/0 (socket says 172.21.15.28:49968) 2024-09-21T05:53:03.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322ffff640 1 -- 172.21.15.28:0/3177617034 learned_addr learned my addr 172.21.15.28:0/3177617034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:03.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322f7fe640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32301029b0 0x7f32301a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 -- 172.21.15.28:0/3177617034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32301029b0 msgr2=0x7f32301a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32301029b0 0x7f32301a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 -- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 msgr2=0x7f32301a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322effd640 1 -- 172.21.15.28:0/3177617034 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32301a6a10 con 0x7f32301082e0 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.011+0000 7f322ffff640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.012+0000 7f322effd640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301a1080 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f322002ea90 tx=0x7f3220004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.012+0000 7f322cff9640 1 -- 172.21.15.28:0/3177617034 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220004030 con 0x7f32301082e0 2024-09-21T05:53:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.012+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32301a6ca0 con 0x7f32301082e0 2024-09-21T05:53:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.012+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32301ad490 con 0x7f32301082e0 2024-09-21T05:53:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.013+0000 7f322cff9640 1 -- 172.21.15.28:0/3177617034 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f322000f3a0 con 0x7f32301082e0 2024-09-21T05:53:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.013+0000 7f322cff9640 1 -- 172.21.15.28:0/3177617034 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220004030 con 0x7f32301082e0 2024-09-21T05:53:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.016+0000 7f323cc92740 1 Processor -- start 2024-09-21T05:53:03.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.017+0000 7f323cc92740 1 -- start start 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa1f99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e334d0 0x55ceaa1fa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa16e6f0 con 0x55cea9e2b2f0 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa15db20 con 0x55cea9e334d0 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f323cc92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa161930 con 0x55cea9e541b0 2024-09-21T05:53:03.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.018+0000 7f322cff9640 1 -- 172.21.15.28:0/3177617034 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f322000c070 con 0x7f32301082e0 2024-09-21T05:53:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.019+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.019+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49982/0 (socket says 172.21.15.28:49982) 2024-09-21T05:53:03.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.019+0000 7f322f7fe640 1 -- 172.21.15.28:0/1561755920 learned_addr learned my addr 172.21.15.28:0/1561755920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322cff9640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 0x7f321c0b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 -- 172.21.15.28:0/1561755920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 msgr2=0x55ceaa1f99b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa1f99b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 -- 172.21.15.28:0/1561755920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e334d0 msgr2=0x55ceaa1fa430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e334d0 0x55ceaa1fa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 -- 172.21.15.28:0/1561755920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceaa123530 con 0x55cea9e2b2f0 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322cff9640 1 -- 172.21.15.28:0/3177617034 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f322000e5d0 con 0x7f32301082e0 2024-09-21T05:53:03.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322f7fe640 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f321800bc30 tx=0x7f3218005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4213f52f52e3c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.020+0000 7f322ffff640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 0x7f321c0b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.022+0000 7f3225ffb640 1 -- 172.21.15.28:0/1561755920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218012ea0 con 0x55cea9e2b2f0 2024-09-21T05:53:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.022+0000 7f3225ffb640 1 -- 172.21.15.28:0/1561755920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321800be40 con 0x55cea9e2b2f0 2024-09-21T05:53:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.024+0000 7f323cc92740 1 -- 172.21.15.28:0/1561755920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 msgr2=0x55ceaa1f9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.024+0000 7f323cc92740 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f321800bc30 tx=0x7f3218005f90 comp rx=0 tx=0).stop 2024-09-21T05:53:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.025+0000 7f322ffff640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 0x7f321c0b8f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f322800c440 tx=0x7f322800b040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.028+0000 7f323cc92740 1 -- 172.21.15.28:0/1561755920 shutdown_connections 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.028+0000 7f323cc92740 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e334d0 0x55ceaa1fa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.028+0000 7f323cc92740 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1f9ef0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.028+0000 7f323cc92740 1 --2- 172.21.15.28:0/1561755920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa1f99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.028+0000 7f323cc92740 1 -- 172.21.15.28:0/1561755920 >> 172.21.15.28:0/1561755920 conn(0x55cea9e48300 msgr2=0x55ceaa1f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.030+0000 7f323cc92740 1 -- 172.21.15.28:0/1561755920 shutdown_connections 2024-09-21T05:53:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.030+0000 7f323cc92740 1 -- 172.21.15.28:0/1561755920 wait complete. 2024-09-21T05:53:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.031+0000 7f323cc92740 1 -- start start 2024-09-21T05:53:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.032+0000 7f323cc92740 1 Processor -- start 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 0x55ceaa215fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 0x55ceaa2164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e541b0 0x55ceaa216a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa095ab0 con 0x55cea9e541b0 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa103730 con 0x55cea9e2b2f0 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f323cc92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ceaa219160 con 0x55cea9e334d0 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 0x55ceaa215fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 0x55ceaa215fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59536/0 (socket says 172.21.15.28:59536) 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f322f7fe640 1 -- 172.21.15.28:0/772258574 learned_addr learned my addr 172.21.15.28:0/772258574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f322effd640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 0x55ceaa2164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.033+0000 7f322ffff640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e541b0 0x55ceaa216a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 -- 172.21.15.28:0/772258574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 msgr2=0x55ceaa215fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 0x55ceaa215fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 -- 172.21.15.28:0/772258574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e541b0 msgr2=0x55ceaa216a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e541b0 0x55ceaa216a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 -- 172.21.15.28:0/772258574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea9ebf790 con 0x55cea9e334d0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f322effd640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 0x55ceaa2164f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3220104ec0 tx=0x7f32200d6ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3cc38e67a292bbf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.034+0000 7f320effd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.035+0000 7f323cc92740 10 client.0 _ll_register_callbacks cb 0x55ceaa1576b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.035+0000 7f320effd640 1 -- 172.21.15.28:0/772258574 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32200c9db0 con 0x55cea9e334d0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.035+0000 7f320effd640 1 -- 172.21.15.28:0/772258574 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f32200d6b50 con 0x55cea9e334d0 2024-09-21T05:53:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.035+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ceaa170640 con 0x55cea9e334d0 2024-09-21T05:53:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.037+0000 7f320effd640 1 -- 172.21.15.28:0/772258574 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f32200c9f50 con 0x55cea9e334d0 2024-09-21T05:53:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.038+0000 7f323cc92740 10 client.26548 fetch_fsmap learned FSMap version 42 2024-09-21T05:53:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.038+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=42}) -- 0x55cea9f38620 con 0x55cea9e334d0 2024-09-21T05:53:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f320effd640 1 -- 172.21.15.28:0/772258574 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 42) ==== 8366+0+0 (secure 0 0 0) 0x7f322004f060 con 0x55cea9e334d0 2024-09-21T05:53:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 10 client.26548 fetch_fsmap finished waiting for FSMap version 42 2024-09-21T05:53:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 10 client.26548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 20 client.26548 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cea9e12d80 0x55ceaa227ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 4 client.26548 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceaa170640 con 0x55cea9e12d80 2024-09-21T05:53:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.040+0000 7f322f7fe640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cea9e12d80 0x55ceaa227ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.042+0000 7f322f7fe640 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cea9e12d80 0x55ceaa227ee0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8618d6d0f91d6082 server_cookie=427cedbb25c4b3da in_seq=0 out_seq=0 2024-09-21T05:53:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.042+0000 7f320effd640 10 client.26548 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.044+0000 7f320effd640 1 -- 172.21.15.28:0/772258574 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ceaa170640 con 0x55cea9e12d80 2024-09-21T05:53:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.044+0000 7f320effd640 10 client.26548 handle_command_reply: tid=0 2024-09-21T05:53:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.045+0000 7f323cc92740 1 client.26548 shutdown 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 msgr2=0x55ceaa2164f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 0x55ceaa2164f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3220104ec0 tx=0x7f32200d6ae0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 shutdown_connections 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55cea9e12d80 0x55ceaa227ee0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e541b0 0x55ceaa216a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e334d0 0x55ceaa2164f0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3220104ec0 tx=0x7f32200d6ae0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 --2- 172.21.15.28:0/772258574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e2b2f0 0x55ceaa215fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 >> 172.21.15.28:0/772258574 conn(0x55cea9e48300 msgr2=0x55ceaa1fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.046+0000 7f320effd640 0 client.26548 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.047+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 shutdown_connections 2024-09-21T05:53:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.047+0000 7f323cc92740 1 -- 172.21.15.28:0/772258574 wait complete. 2024-09-21T05:53:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.047+0000 7f323cc92740 20 client.26548 trim_cache size 0 max 16384 2024-09-21T05:53:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.075+0000 7f323cc92740 1 Processor -- start 2024-09-21T05:53:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.075+0000 7f323cc92740 1 -- start start 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.075+0000 7f323cc92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.075+0000 7f323cc92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa214810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.075+0000 7f323cc92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9de81c0 0x55ceaa214d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f323cc92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f321000e810 con 0x55cea9e2b2f0 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f323cc92740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3210010380 con 0x55cea9de81c0 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f323cc92740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f321000eee0 con 0x55cea9e541b0 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50004/0 (socket says 172.21.15.28:50004) 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 -- 172.21.15.28:0/721360491 learned_addr learned my addr 172.21.15.28:0/721360491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322effd640 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa214810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322ffff640 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9de81c0 0x55ceaa214d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 -- 172.21.15.28:0/721360491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 msgr2=0x55ceaa214810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa214810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 -- 172.21.15.28:0/721360491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9de81c0 msgr2=0x55ceaa214d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9de81c0 0x55ceaa214d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 -- 172.21.15.28:0/721360491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea9e12fb0 con 0x55cea9e2b2f0 2024-09-21T05:53:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.076+0000 7f322f7fe640 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f321801ebb0 tx=0x7f321801e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb0685a04679c84 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.077+0000 7f32257fa640 1 -- 172.21.15.28:0/721360491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321802d030 con 0x55cea9e2b2f0 2024-09-21T05:53:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.077+0000 7f32257fa640 1 -- 172.21.15.28:0/721360491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321800b620 con 0x55cea9e2b2f0 2024-09-21T05:53:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.077+0000 7f32257fa640 1 -- 172.21.15.28:0/721360491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218003c60 con 0x55cea9e2b2f0 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 -- 172.21.15.28:0/721360491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 msgr2=0x55ceaa227df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f321801ebb0 tx=0x7f321801e5b0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 -- 172.21.15.28:0/721360491 shutdown_connections 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9de81c0 0x55ceaa214d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9e541b0 0x55ceaa214810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 --2- 172.21.15.28:0/721360491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa227df0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 -- 172.21.15.28:0/721360491 >> 172.21.15.28:0/721360491 conn(0x55cea9e48300 msgr2=0x55ceaa1f7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 -- 172.21.15.28:0/721360491 shutdown_connections 2024-09-21T05:53:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.078+0000 7f323cc92740 1 -- 172.21.15.28:0/721360491 wait complete. 2024-09-21T05:53:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.079+0000 7f323cc92740 1 -- start start 2024-09-21T05:53:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.079+0000 7f323cc92740 1 Processor -- start 2024-09-21T05:53:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9de81c0 0x55ceaa21ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1fa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 0x55ceaa1fad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f322effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1fa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f322effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1fa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50010/0 (socket says 172.21.15.28:50010) 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f322effd640 1 -- 172.21.15.28:0/999046005 learned_addr learned my addr 172.21.15.28:0/999046005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f321000fcb0 con 0x55cea9e2b2f0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f321000f5b0 con 0x55cea9e541b0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f321000c7f0 con 0x55cea9de81c0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f322f7fe640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9de81c0 0x55ceaa21ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f322ffff640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 0x55ceaa1fad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.080+0000 7f323cc92740 10 client.0 _ll_register_callbacks cb 0x7f32280095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 -- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9de81c0 msgr2=0x55ceaa21ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9de81c0 0x55ceaa21ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 -- 172.21.15.28:0/999046005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 msgr2=0x55ceaa1fa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1fa7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceaa1fb470 con 0x55cea9e541b0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f322ffff640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 0x55ceaa1fad30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3228009fd0 tx=0x7f3228004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c38cf35a2748bf17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.081+0000 7f320effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.082+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228025030 con 0x55cea9e541b0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.082+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f321800b620 con 0x55cea9e541b0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.082+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f322802a410 con 0x55cea9e541b0 2024-09-21T05:53:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.082+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f322801b040 con 0x55cea9e541b0 2024-09-21T05:53:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.084+0000 7f323cc92740 10 client.26552 fetch_fsmap learned FSMap version 42 2024-09-21T05:53:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.084+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ceaa2928e0 con 0x55cea9e541b0 2024-09-21T05:53:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.086+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321800b620 con 0x55cea9e541b0 2024-09-21T05:53:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 42) ==== 8366+0+0 (secure 0 0 0) 0x7f3228031020 con 0x55cea9e541b0 2024-09-21T05:53:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 10 client.26552 fetch_fsmap finished waiting for FSMap version 42 2024-09-21T05:53:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 10 client.26552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 20 client.26552 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ceaa292fe0 0x55ceaa295460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 4 client.26552 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:53:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ceaa16c0f0 con 0x55ceaa292fe0 2024-09-21T05:53:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.087+0000 7f322f7fe640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ceaa292fe0 0x55ceaa295460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.088+0000 7f322f7fe640 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ceaa292fe0 0x55ceaa295460 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51be321d74a0ef79 server_cookie=29fc066a9a2308a6 in_seq=0 out_seq=0 2024-09-21T05:53:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.088+0000 7f320effd640 10 client.26552 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f320effd640 1 -- 172.21.15.28:0/999046005 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ceaa16c0f0 con 0x55ceaa292fe0 2024-09-21T05:53:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f320effd640 10 client.26552 handle_command_reply: tid=0 2024-09-21T05:53:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 client.26552 shutdown 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 msgr2=0x55ceaa1fad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 0x55ceaa1fad30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3228009fd0 tx=0x7f3228004b50 comp rx=0 tx=0).stop 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 shutdown_connections 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55ceaa292fe0 0x55ceaa295460 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea9e541b0 0x55ceaa1fad30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cea9e2b2f0 0x55ceaa1fa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 --2- 172.21.15.28:0/999046005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cea9de81c0 0x55ceaa21ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.119+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 >> 172.21.15.28:0/999046005 conn(0x55cea9e48300 msgr2=0x55ceaa1f6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.120+0000 7f320effd640 0 client.26552 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.120+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 shutdown_connections 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.120+0000 7f323cc92740 1 -- 172.21.15.28:0/999046005 wait complete. 2024-09-21T05:53:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.120+0000 7f323cc92740 20 client.26552 trim_cache size 0 max 16384 2024-09-21T05:53:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:53:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:53:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:53:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:53:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 msgr2=0x7f321c0b8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 0x7f321c0b8f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f322800c440 tx=0x7f322800b040 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 msgr2=0x7f32301a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301a1080 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f322002ea90 tx=0x7f3220004970 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 shutdown_connections 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f321c0b6a80 0x7f321c0b8f90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3230108c30 0x7f32301a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32301082e0 0x7f32301a1080 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 --2- 172.21.15.28:0/3177617034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32301029b0 0x7f32301a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 >> 172.21.15.28:0/3177617034 conn(0x7f323007a740 msgr2=0x7f32301063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 shutdown_connections 2024-09-21T05:53:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:03.121+0000 7f3235d20640 1 -- 172.21.15.28:0/3177617034 wait complete. 2024-09-21T05:53:04.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:03 smithi181 ceph-mon[34819]: pgmap v506: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 105 KiB/s rd, 35 MiB/s wr, 8.02k op/s 2024-09-21T05:53:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:03 smithi028 ceph-mon[31227]: pgmap v506: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 105 KiB/s rd, 35 MiB/s wr, 8.02k op/s 2024-09-21T05:53:04.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:03 smithi129 ceph-mon[34885]: pgmap v506: 97 pgs: 97 active+clean; 17 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 105 KiB/s rd, 35 MiB/s wr, 8.02k op/s 2024-09-21T05:53:06.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:05 smithi181 ceph-mon[34819]: pgmap v507: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 7.18k op/s 2024-09-21T05:53:06.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:05 smithi028 ceph-mon[31227]: pgmap v507: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 7.18k op/s 2024-09-21T05:53:06.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:05 smithi129 ceph-mon[34885]: pgmap v507: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 7.18k op/s 2024-09-21T05:53:08.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:07 smithi181 ceph-mon[34819]: pgmap v508: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 68 KiB/s rd, 28 MiB/s wr, 6.49k op/s 2024-09-21T05:53:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:07 smithi028 ceph-mon[31227]: pgmap v508: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 68 KiB/s rd, 28 MiB/s wr, 6.49k op/s 2024-09-21T05:53:08.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:07 smithi129 ceph-mon[34885]: pgmap v508: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 68 KiB/s rd, 28 MiB/s wr, 6.49k op/s 2024-09-21T05:53:10.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:09 smithi181 ceph-mon[34819]: pgmap v509: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 73 KiB/s rd, 19 MiB/s wr, 4.46k op/s 2024-09-21T05:53:10.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:09 smithi028 ceph-mon[31227]: pgmap v509: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 73 KiB/s rd, 19 MiB/s wr, 4.46k op/s 2024-09-21T05:53:10.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:09 smithi129 ceph-mon[34885]: pgmap v509: 97 pgs: 97 active+clean; 17 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 73 KiB/s rd, 19 MiB/s wr, 4.46k op/s 2024-09-21T05:53:12.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:11 smithi181 ceph-mon[34819]: pgmap v510: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.98k op/s 2024-09-21T05:53:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:11 smithi028 ceph-mon[31227]: pgmap v510: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.98k op/s 2024-09-21T05:53:12.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:11 smithi129 ceph-mon[34885]: pgmap v510: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.98k op/s 2024-09-21T05:53:14.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:13 smithi181 ceph-mon[34819]: pgmap v511: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 58 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2024-09-21T05:53:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:13 smithi028 ceph-mon[31227]: pgmap v511: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 58 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2024-09-21T05:53:14.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:13 smithi129 ceph-mon[34885]: pgmap v511: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 58 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2024-09-21T05:53:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:15 smithi181 ceph-mon[34819]: pgmap v512: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.91k op/s 2024-09-21T05:53:16.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:15 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:53:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.297+0000 7f77e6fe4640 1 -- 172.21.15.28:0/551779171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0104920 msgr2=0x7f77e0104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.297+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/551779171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0104920 0x7f77e0104d30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009ca0 tx=0x7f77d802a8e0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.305+0000 7f77e6fe4640 1 -- 172.21.15.28:0/551779171 shutdown_connections 2024-09-21T05:53:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.305+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/551779171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e010dd70 0x7f77e0110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.305+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/551779171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e0105270 0x7f77e010d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.305+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/551779171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0104920 0x7f77e0104d30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.305+0000 7f77e6fe4640 1 -- 172.21.15.28:0/551779171 >> 172.21.15.28:0/551779171 conn(0x7f77e007a6f0 msgr2=0x7f77e007ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.307+0000 7f77e6fe4640 1 -- 172.21.15.28:0/551779171 shutdown_connections 2024-09-21T05:53:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.307+0000 7f77e6fe4640 1 -- 172.21.15.28:0/551779171 wait complete. 2024-09-21T05:53:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 Processor -- start 2024-09-21T05:53:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 -- start start 2024-09-21T05:53:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:15 smithi028 ceph-mon[31227]: pgmap v512: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.91k op/s 2024-09-21T05:53:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:15 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e0104920 0x7f77e0102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e010dd70 0x7f77e01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77e006d6e0 con 0x7f77e010dd70 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77e006d560 con 0x7f77e0105270 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.309+0000 7f77e6fe4640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77e006d860 con 0x7f77e0104920 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.310+0000 7f77dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.310+0000 7f77dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45674/0 (socket says 172.21.15.28:45674) 2024-09-21T05:53:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.310+0000 7f77dffff640 1 -- 172.21.15.28:0/1563175069 learned_addr learned my addr 172.21.15.28:0/1563175069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e0104920 0x7f77e0102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77dffff640 1 -- 172.21.15.28:0/1563175069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e0104920 msgr2=0x7f77e0102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77dffff640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e0104920 0x7f77e0102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77dffff640 1 -- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e010dd70 msgr2=0x7f77e01030b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77dffff640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e010dd70 0x7f77e01030b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.311+0000 7f77dffff640 1 -- 172.21.15.28:0/1563175069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e010d950 con 0x7f77e0105270 2024-09-21T05:53:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.315+0000 7f77dffff640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f77d000e9a0 tx=0x7f77d000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.320+0000 7f77ddffb640 1 -- 172.21.15.28:0/1563175069 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d000cd80 con 0x7f77e0105270 2024-09-21T05:53:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.320+0000 7f77ddffb640 1 -- 172.21.15.28:0/1563175069 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d00035f0 con 0x7f77e0105270 2024-09-21T05:53:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.320+0000 7f77ddffb640 1 -- 172.21.15.28:0/1563175069 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0010690 con 0x7f77e0105270 2024-09-21T05:53:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.324+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e01b56f0 con 0x7f77e0105270 2024-09-21T05:53:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.324+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e01b5b80 con 0x7f77e0105270 2024-09-21T05:53:16.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.331+0000 7f77ddffb640 1 -- 172.21.15.28:0/1563175069 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f77d0003fb0 con 0x7f77e0105270 2024-09-21T05:53:16.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.335+0000 7f77ddffb640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 0x7f77cc0b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.336+0000 7f77ddffb640 1 -- 172.21.15.28:0/1563175069 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f77d00e7af0 con 0x7f77e0105270 2024-09-21T05:53:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.336+0000 7f77ee031740 1 Processor -- start 2024-09-21T05:53:16.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.339+0000 7f77ee031740 1 -- start start 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caec69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 0x5641caec6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641cae3b6f0 con 0x5641caaf82f0 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5641cae2ab20 con 0x5641cab004d0 2024-09-21T05:53:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.340+0000 7f77ee031740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5641cae2e930 con 0x5641cab211b0 2024-09-21T05:53:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.341+0000 7f77e5581640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.342+0000 7f77e5581640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45686/0 (socket says 172.21.15.28:45686) 2024-09-21T05:53:16.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.342+0000 7f77e5581640 1 -- 172.21.15.28:0/3332261784 learned_addr learned my addr 172.21.15.28:0/3332261784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:16.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.342+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 0x7f77cc0b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.346+0000 7f77dffff640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 0x5641caec6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.347+0000 7f77e5581640 1 -- 172.21.15.28:0/3332261784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 msgr2=0x5641caec69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.349+0000 7f77e4d80640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caec69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.351+0000 7f77e5581640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caec69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.351+0000 7f77e5581640 1 -- 172.21.15.28:0/3332261784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 msgr2=0x5641caec6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.351+0000 7f77e5581640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 0x5641caec6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.351+0000 7f77e5581640 1 -- 172.21.15.28:0/3332261784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641cadf0530 con 0x5641cab004d0 2024-09-21T05:53:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.354+0000 7f77e4d80640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caec69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.356+0000 7f77dffff640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 0x5641caec6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:53:16.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.357+0000 7f77e5581640 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009720 tx=0x7f77c8009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2acff7f2cc3a2c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.358+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 0x7f77cc0b8d90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000f80 tx=0x7f77d8009780 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:15 smithi129 ceph-mon[34885]: pgmap v512: 97 pgs: 97 active+clean; 17 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.91k op/s 2024-09-21T05:53:16.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:15 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.360+0000 7f77d6ffd640 1 -- 172.21.15.28:0/3332261784 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c8015020 con 0x5641cab004d0 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.360+0000 7f77d6ffd640 1 -- 172.21.15.28:0/3332261784 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c8009e30 con 0x5641cab004d0 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.360+0000 7f77d6ffd640 1 -- 172.21.15.28:0/3332261784 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c8010380 con 0x5641cab004d0 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.362+0000 7f77ee031740 1 -- 172.21.15.28:0/3332261784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 msgr2=0x5641caec7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.363+0000 7f77ee031740 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009720 tx=0x7f77c8009be0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.365+0000 7f77ee031740 1 -- 172.21.15.28:0/3332261784 shutdown_connections 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.365+0000 7f77ee031740 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caec7430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.365+0000 7f77ee031740 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641caaf82f0 0x5641caec6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.365+0000 7f77ee031740 1 --2- 172.21.15.28:0/3332261784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caec69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.365+0000 7f77ee031740 1 -- 172.21.15.28:0/3332261784 >> 172.21.15.28:0/3332261784 conn(0x5641cab15300 msgr2=0x5641caec48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.372+0000 7f77ee031740 1 -- 172.21.15.28:0/3332261784 shutdown_connections 2024-09-21T05:53:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.375+0000 7f77ee031740 1 -- 172.21.15.28:0/3332261784 wait complete. 2024-09-21T05:53:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 -- start start 2024-09-21T05:53:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 Processor -- start 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab004d0 0x5641caed2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caeef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641cad62ab0 con 0x5641cab004d0 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5641cadd0730 con 0x5641caaf82f0 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.376+0000 7f77ee031740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5641caee6140 con 0x5641cab211b0 2024-09-21T05:53:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.377+0000 7f77ee031740 10 client.0 _ll_register_callbacks cb 0x5641cae246b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.377+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.377+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45692/0 (socket says 172.21.15.28:45692) 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.377+0000 7f77e4d80640 1 -- 172.21.15.28:0/1917002050 learned_addr learned my addr 172.21.15.28:0/1917002050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.377+0000 7f77e5581640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caeef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 -- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 msgr2=0x5641caeef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caeef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 -- 172.21.15.28:0/1917002050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab004d0 msgr2=0x5641caed2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab004d0 0x5641caed2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 -- 172.21.15.28:0/1917002050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641cab8c790 con 0x5641caaf82f0 2024-09-21T05:53:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.378+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77d8035000 tx=0x7f77d8009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7c29688114adb2d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.379+0000 7f77bffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.379+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d804d070 con 0x5641caaf82f0 2024-09-21T05:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.380+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c8009e30 con 0x5641caaf82f0 2024-09-21T05:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.380+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f77d80047c0 con 0x5641caaf82f0 2024-09-21T05:53:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.382+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f77d8057050 con 0x5641caaf82f0 2024-09-21T05:53:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.382+0000 7f77ee031740 10 client.26564 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.382+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5641cac05620 con 0x5641caaf82f0 2024-09-21T05:53:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c8009e30 con 0x5641caaf82f0 2024-09-21T05:53:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f77d8048040 con 0x5641caaf82f0 2024-09-21T05:53:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 10 client.26564 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 10 client.26564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 20 client.26564 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caadfd80 0x5641caef5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 4 client.26564 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.389+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641cae3d640 con 0x5641caadfd80 2024-09-21T05:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.390+0000 7f77dffff640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caadfd80 0x5641caef5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.406+0000 7f77dffff640 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caadfd80 0x5641caef5740 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2233d1f2676a2c55 server_cookie=d983cd24d216ab4d in_seq=0 out_seq=0 2024-09-21T05:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.407+0000 7f77bffff640 10 client.26564 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.411+0000 7f77bffff640 1 -- 172.21.15.28:0/1917002050 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5641cae3d640 con 0x5641caadfd80 2024-09-21T05:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.411+0000 7f77bffff640 10 client.26564 handle_command_reply: tid=0 2024-09-21T05:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.411+0000 7f77ee031740 1 client.26564 shutdown 2024-09-21T05:53:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.419+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 msgr2=0x5641caed2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.419+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77d8035000 tx=0x7f77d8009440 comp rx=0 tx=0).stop 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 shutdown_connections 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caadfd80 0x5641caef5740 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab211b0 0x5641caeef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab004d0 0x5641caed2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 --2- 172.21.15.28:0/1917002050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed2110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 >> 172.21.15.28:0/1917002050 conn(0x5641cab15300 msgr2=0x5641caec79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.420+0000 7f77bffff640 0 client.26564 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.425+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 shutdown_connections 2024-09-21T05:53:16.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.426+0000 7f77ee031740 1 -- 172.21.15.28:0/1917002050 wait complete. 2024-09-21T05:53:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.427+0000 7f77ee031740 20 client.26564 trim_cache size 0 max 16384 2024-09-21T05:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.470+0000 7f77ee031740 1 Processor -- start 2024-09-21T05:53:16.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.472+0000 7f77ee031740 1 -- start start 2024-09-21T05:53:16.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.472+0000 7f77ee031740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77ee031740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab004d0 0x5641caeef330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45698/0 (socket says 172.21.15.28:45698) 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77e4d80640 1 -- 172.21.15.28:0/225823508 learned_addr learned my addr 172.21.15.28:0/225823508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77ee031740 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caec3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01b5ea0 con 0x5641cab211b0 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01b7a40 con 0x5641caaf82f0 2024-09-21T05:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.473+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01b6570 con 0x5641cab004d0 2024-09-21T05:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.474+0000 7f77e4d80640 1 -- 172.21.15.28:0/225823508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab004d0 msgr2=0x5641caeef330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.474+0000 7f77e4d80640 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab004d0 0x5641caeef330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.474+0000 7f77e4d80640 1 -- 172.21.15.28:0/225823508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 msgr2=0x5641caec3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.474+0000 7f77e4d80640 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caec3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.474+0000 7f77e4d80640 1 -- 172.21.15.28:0/225823508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641caed05f0 con 0x5641caaf82f0 2024-09-21T05:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.475+0000 7f77e4d80640 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009ca0 tx=0x7f77d8009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84a17b42f90087f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.476+0000 7f77d67fc640 1 -- 172.21.15.28:0/225823508 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d804d040 con 0x5641caaf82f0 2024-09-21T05:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.476+0000 7f77d67fc640 1 -- 172.21.15.28:0/225823508 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d8036040 con 0x5641caaf82f0 2024-09-21T05:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.476+0000 7f77d67fc640 1 -- 172.21.15.28:0/225823508 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8042040 con 0x5641caaf82f0 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 msgr2=0x5641caed3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009ca0 tx=0x7f77d8009360 comp rx=0 tx=0).stop 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 shutdown_connections 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caec3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641cab004d0 0x5641caeef330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 --2- 172.21.15.28:0/225823508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641caaf82f0 0x5641caed3d40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 >> 172.21.15.28:0/225823508 conn(0x5641cab15300 msgr2=0x5641caed3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 shutdown_connections 2024-09-21T05:53:16.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.478+0000 7f77ee031740 1 -- 172.21.15.28:0/225823508 wait complete. 2024-09-21T05:53:16.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.479+0000 7f77ee031740 1 -- start start 2024-09-21T05:53:16.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.479+0000 7f77ee031740 1 Processor -- start 2024-09-21T05:53:16.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.480+0000 7f77ee031740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.480+0000 7f77ee031740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caee6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77e4d80640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:43900/0 (socket says 172.21.15.28:43900) 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caee6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caee6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45702/0 (socket says 172.21.15.28:45702) 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77ee031740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caee6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77ee031740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01b7370 con 0x5641cab211b0 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77ee031740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01b6c40 con 0x5641cab004d0 2024-09-21T05:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77ee031740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77e01af6a0 con 0x5641caaf82f0 2024-09-21T05:53:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77ee031740 10 client.0 _ll_register_callbacks cb 0x7f77d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.481+0000 7f77e4d80640 1 -- 172.21.15.28:0/1596710053 learned_addr learned my addr 172.21.15.28:0/1596710053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e5581640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caee6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 -- 172.21.15.28:0/1596710053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 msgr2=0x5641caee6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caee6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 -- 172.21.15.28:0/1596710053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 msgr2=0x5641caee6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caee6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 -- 172.21.15.28:0/1596710053 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641caf5e2e0 con 0x5641caaf82f0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e5581640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caee6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77e4d80640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77d80046c0 tx=0x7f77d80046f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed06e5b30b92a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77bffff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.482+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8042040 con 0x5641caaf82f0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.483+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f77d80309f0 con 0x5641caaf82f0 2024-09-21T05:53:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.483+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77d8036040 con 0x5641caaf82f0 2024-09-21T05:53:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.485+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f77d804d040 con 0x5641caaf82f0 2024-09-21T05:53:16.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.487+0000 7f77ee031740 10 client.26563 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.487+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5641caf5f570 con 0x5641caaf82f0 2024-09-21T05:53:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.489+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d8036040 con 0x5641caaf82f0 2024-09-21T05:53:16.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.490+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f77d8048070 con 0x5641caaf82f0 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 10 client.26563 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 10 client.26563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 20 client.26563 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caab51c0 0x5641caf61c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 4 client.26563 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:53:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.492+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641cae390f0 con 0x5641caab51c0 2024-09-21T05:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.493+0000 7f77dffff640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caab51c0 0x5641caf61c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.496+0000 7f77dffff640 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caab51c0 0x5641caf61c70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c962de844e753d37 server_cookie=bbf21a9e379d9b4b in_seq=0 out_seq=0 2024-09-21T05:53:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.496+0000 7f77bffff640 10 client.26563 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.559+0000 7f77bffff640 1 -- 172.21.15.28:0/1596710053 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5641cae390f0 con 0x5641caab51c0 2024-09-21T05:53:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.559+0000 7f77bffff640 10 client.26563 handle_command_reply: tid=0 2024-09-21T05:53:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.559+0000 7f77ee031740 1 client.26563 shutdown 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.559+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 msgr2=0x5641caee6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.559+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77d80046c0 tx=0x7f77d80046f0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 shutdown_connections 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5641caab51c0 0x5641caf61c70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77bffff640 0 client.26563 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641cab211b0 0x5641caee6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641cab004d0 0x5641caee6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 --2- 172.21.15.28:0/1596710053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641caaf82f0 0x5641caee6500 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 >> 172.21.15.28:0/1596710053 conn(0x5641cab15300 msgr2=0x5641caed35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 shutdown_connections 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.560+0000 7f77ee031740 1 -- 172.21.15.28:0/1596710053 wait complete. 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.561+0000 7f77ee031740 20 client.26563 trim_cache size 0 max 16384 2024-09-21T05:53:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:53:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:53:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:53:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.562+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 msgr2=0x7f77cc0b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.562+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 0x7f77cc0b8d90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000f80 tx=0x7f77d8009780 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.562+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 msgr2=0x7f77e0102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.562+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f77d000e9a0 tx=0x7f77d000ee60 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 shutdown_connections 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f77cc0b6880 0x7f77cc0b8d90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e010dd70 0x7f77e01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e0105270 0x7f77e0102b70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 --2- 172.21.15.28:0/1563175069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77e0104920 0x7f77e0102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 >> 172.21.15.28:0/1563175069 conn(0x7f77e007a6f0 msgr2=0x7f77e0100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 shutdown_connections 2024-09-21T05:53:16.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:16.563+0000 7f77e6fe4640 1 -- 172.21.15.28:0/1563175069 wait complete. 2024-09-21T05:53:18.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:17 smithi181 ceph-mon[34819]: pgmap v513: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-09-21T05:53:18.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:53:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:17 smithi028 ceph-mon[31227]: pgmap v513: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-09-21T05:53:18.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:17 smithi129 ceph-mon[34885]: pgmap v513: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 4.96k op/s 2024-09-21T05:53:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.385+0000 7f7877586640 1 -- 172.21.15.28:0/495815476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680ad480 msgr2=0x7f78680b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.385+0000 7f7875582640 1 -- 172.21.15.28:0/495815476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787006c270 con 0x7f78680ad480 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.385+0000 7f7877586640 1 --2- 172.21.15.28:0/495815476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680ad480 0x7f78680b3c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7870066480 tx=0x7f787008f960 comp rx=0 tx=0).stop 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.386+0000 7f7877586640 1 -- 172.21.15.28:0/495815476 shutdown_connections 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.386+0000 7f7877586640 1 --2- 172.21.15.28:0/495815476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680ad480 0x7f78680b3c20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.386+0000 7f7877586640 1 --2- 172.21.15.28:0/495815476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680acb30 0x7f78680acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.386+0000 7f7877586640 1 --2- 172.21.15.28:0/495815476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680a50f0 0x7f78680a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.386+0000 7f7877586640 1 -- 172.21.15.28:0/495815476 >> 172.21.15.28:0/495815476 conn(0x7f786801cc70 msgr2=0x7f786801d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.387+0000 7f7877586640 1 -- 172.21.15.28:0/495815476 shutdown_connections 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.387+0000 7f7877586640 1 -- 172.21.15.28:0/495815476 wait complete. 2024-09-21T05:53:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.388+0000 7f7877586640 1 Processor -- start 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.388+0000 7f7877586640 1 -- start start 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 0x7f78680aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680ad480 0x7f78680af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78680b7090 con 0x7f78680a50f0 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78680b6f10 con 0x7f78680ad480 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7877586640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f78680b7210 con 0x7f78680acb30 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53132/0 (socket says 172.21.15.28:53132) 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7876584640 1 -- 172.21.15.28:0/3633146907 learned_addr learned my addr 172.21.15.28:0/3633146907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7875d83640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 0x7f78680aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.389+0000 7f7876d85640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680ad480 0x7f78680af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 -- 172.21.15.28:0/3633146907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 msgr2=0x7f78680aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 0x7f78680aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 -- 172.21.15.28:0/3633146907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680ad480 msgr2=0x7f78680af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680ad480 0x7f78680af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 -- 172.21.15.28:0/3633146907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78681479b0 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7875d83640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 0x7f78680aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.390+0000 7f7876584640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f786c00b450 tx=0x7f786c00b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.391+0000 7f78677fe640 1 -- 172.21.15.28:0/3633146907 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c015020 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.391+0000 7f78677fe640 1 -- 172.21.15.28:0/3633146907 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f786c011ce0 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.391+0000 7f78677fe640 1 -- 172.21.15.28:0/3633146907 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c010dc0 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.391+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7868147ca0 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.391+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7868148190 con 0x7f78680a50f0 2024-09-21T05:53:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.395+0000 7f787ed5c740 1 Processor -- start 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.395+0000 7f787ed5c740 1 -- start start 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785de8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785de8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564785d5ee50 con 0x564785a181b0 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564785d4e280 con 0x564785a20390 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f787ed5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564785d52090 con 0x564785a41070 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f78677fe640 1 -- 172.21.15.28:0/3633146907 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f786c018020 con 0x7f78680a50f0 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f7876d85640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785de8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f7875d83640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785de8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f7875d83640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45730/0 (socket says 172.21.15.28:45730) 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.396+0000 7f7875d83640 1 -- 172.21.15.28:0/3708913894 learned_addr learned my addr 172.21.15.28:0/3708913894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.397+0000 7f78677fe640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 0x7f785c0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 -- 172.21.15.28:0/3708913894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 msgr2=0x564785de8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785de8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 -- 172.21.15.28:0/3708913894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 msgr2=0x564785de8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785de8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 -- 172.21.15.28:0/3708913894 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564785aac650 con 0x564785a20390 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f78677fe640 1 -- 172.21.15.28:0/3633146907 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f786c0e6e50 con 0x7f78680a50f0 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7876d85640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 0x7f785c0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7875d83640 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f786000f3c0 tx=0x7f786000f880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=513d18d23f5e3e47 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7864ff9640 1 -- 172.21.15.28:0/3708913894 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860005220 con 0x564785a20390 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.398+0000 7f7864ff9640 1 -- 172.21.15.28:0/3708913894 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7860004c50 con 0x564785a20390 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.399+0000 7f7864ff9640 1 -- 172.21.15.28:0/3708913894 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860010bb0 con 0x564785a20390 2024-09-21T05:53:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- 172.21.15.28:0/3708913894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 msgr2=0x564785de8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f786000f3c0 tx=0x7f786000f880 comp rx=0 tx=0).stop 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f7876d85640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 0x7f785c0b8c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7870052860 tx=0x7f78700514d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- 172.21.15.28:0/3708913894 shutdown_connections 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785de8e90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785de8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3708913894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785de8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- 172.21.15.28:0/3708913894 >> 172.21.15.28:0/3708913894 conn(0x564785a35460 msgr2=0x564785de6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- 172.21.15.28:0/3708913894 shutdown_connections 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- 172.21.15.28:0/3708913894 wait complete. 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.400+0000 7f787ed5c740 1 -- start start 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 Processor -- start 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785df3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785df4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564785baf700 con 0x564785a181b0 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564785ce9110 con 0x564785a20390 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564785df6c90 con 0x564785a41070 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53158/0 (socket says 172.21.15.28:53158) 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f7876584640 1 -- 172.21.15.28:0/2956269738 learned_addr learned my addr 172.21.15.28:0/2956269738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f787ed5c740 10 client.0 _ll_register_callbacks cb 0x564785d02990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.401+0000 7f7876d85640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785df4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 -- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 msgr2=0x564785df4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785df4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 -- 172.21.15.28:0/2956269738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 msgr2=0x564785df3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785df3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 -- 172.21.15.28:0/2956269738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564785d16c40 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876d85640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785df4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7876584640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f786c009fd0 tx=0x7f786c0a8c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0ba34963f029d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c009280 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.402+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7860004c50 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.403+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f786c0d00e0 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.403+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f786c0a83d0 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.403+0000 7f787ed5c740 10 client.16986 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.403+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564785e02910 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.403+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f786c0b1020 con 0x564785a181b0 2024-09-21T05:53:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f786c0b11d0 con 0x564785a181b0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 10 client.16986 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 20 client.16986 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5647859ff9f0 0x564785e05e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 4 client.16986 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7860004c50 con 0x5647859ff9f0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.404+0000 7f7875d83640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5647859ff9f0 0x564785e05e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.405+0000 7f7875d83640 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5647859ff9f0 0x564785e05e10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6c2b7c6b857614e server_cookie=424a87a54cf5ef4a in_seq=0 out_seq=0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.405+0000 7f7859ffb640 10 client.16986 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.406+0000 7f7859ffb640 1 -- 172.21.15.28:0/2956269738 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f7860004c50 con 0x5647859ff9f0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.406+0000 7f7859ffb640 10 client.16986 handle_command_reply: tid=0 2024-09-21T05:53:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.406+0000 7f787ed5c740 1 client.16986 shutdown 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.406+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 msgr2=0x564785df3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.406+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f786c009fd0 tx=0x7f786c0a8c30 comp rx=0 tx=0).stop 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 shutdown_connections 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5647859ff9f0 0x564785e05e10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f7859ffb640 0 client.16986 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a41070 0x564785df4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a20390 0x564785df3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 --2- 172.21.15.28:0/2956269738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564785a181b0 0x564785df3990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 >> 172.21.15.28:0/2956269738 conn(0x564785a35460 msgr2=0x564785deb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 shutdown_connections 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.407+0000 7f787ed5c740 1 -- 172.21.15.28:0/2956269738 wait complete. 2024-09-21T05:53:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.408+0000 7f787ed5c740 20 client.16986 trim_cache size 0 max 16384 2024-09-21T05:53:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.435+0000 7f787ed5c740 1 Processor -- start 2024-09-21T05:53:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 -- start start 2024-09-21T05:53:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785de5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785de5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53160/0 (socket says 172.21.15.28:53160) 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f7876584640 1 -- 172.21.15.28:0/3941022265 learned_addr learned my addr 172.21.15.28:0/3941022265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78681619a0 con 0x5647859ff9f0 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7868163540 con 0x564785a181b0 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.436+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7868162070 con 0x564785a20390 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876d85640 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785de5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 -- 172.21.15.28:0/3941022265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 msgr2=0x564785de5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785de5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 -- 172.21.15.28:0/3941022265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 msgr2=0x564785de5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785de5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 -- 172.21.15.28:0/3941022265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564785df5d60 con 0x5647859ff9f0 2024-09-21T05:53:19.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f7876584640 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f786c0a9230 tx=0x7f786c0107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b503f53f5d2d10 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f785bfff640 1 -- 172.21.15.28:0/3941022265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c0ac770 con 0x5647859ff9f0 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f785bfff640 1 -- 172.21.15.28:0/3941022265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f786c0b1030 con 0x5647859ff9f0 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.437+0000 7f785bfff640 1 -- 172.21.15.28:0/3941022265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c0b11e0 con 0x5647859ff9f0 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 msgr2=0x564785df4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f786c0a9230 tx=0x7f786c0107e0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 shutdown_connections 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785de5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785de5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3941022265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785df4eb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 >> 172.21.15.28:0/3941022265 conn(0x564785a35460 msgr2=0x564785de9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 shutdown_connections 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.438+0000 7f787ed5c740 1 -- 172.21.15.28:0/3941022265 wait complete. 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.439+0000 7f787ed5c740 1 -- start start 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.439+0000 7f787ed5c740 1 Processor -- start 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.439+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785df7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785df81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7868162e70 con 0x5647859ff9f0 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7868162740 con 0x564785a181b0 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7875d83640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785df7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53174/0 (socket says 172.21.15.28:53174) 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 -- 172.21.15.28:0/3891055648 learned_addr learned my addr 172.21.15.28:0/3891055648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f786815f980 con 0x564785a20390 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876d85640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785df81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f787ed5c740 10 client.0 _ll_register_callbacks cb 0x7f786c0ea5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 -- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 msgr2=0x564785df81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785df81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 -- 172.21.15.28:0/3891055648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 msgr2=0x564785df7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785df7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.440+0000 7f7876584640 1 -- 172.21.15.28:0/3891055648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564785df8940 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7876584640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f786c0b1650 tx=0x7f786c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a698407a1103c957 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c0b13e0 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786c0b1030 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f786c0abec0 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f786c0ac770 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f787ed5c740 10 client.16992 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564785e70720 con 0x5647859ff9f0 2024-09-21T05:53:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.441+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f786c0b1030 con 0x5647859ff9f0 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f786c0ab930 con 0x5647859ff9f0 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 10 client.16992 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 20 client.16992 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564785a41070 0x564785e72c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 4 client.16992 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564785d60da0 con 0x564785a41070 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.442+0000 7f7875d83640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564785a41070 0x564785e72c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f7875d83640 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564785a41070 0x564785e72c90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23a51229c603245b server_cookie=cec93658bcfc63ae in_seq=0 out_seq=0 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f7859ffb640 10 client.16992 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f7859ffb640 1 -- 172.21.15.28:0/3891055648 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2646 (crc 0 0 0) 0x564785d60da0 con 0x564785a41070 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f7859ffb640 10 client.16992 handle_command_reply: tid=0 2024-09-21T05:53:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 client.16992 shutdown 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 msgr2=0x564785dfc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f786c0b1650 tx=0x7f786c0046c0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 shutdown_connections 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x564785a41070 0x564785e72c90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564785a20390 0x564785df81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f7859ffb640 0 client.16992 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564785a181b0 0x564785df7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 --2- 172.21.15.28:0/3891055648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647859ff9f0 0x564785dfc7a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 >> 172.21.15.28:0/3891055648 conn(0x564785a35460 msgr2=0x564785dff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 shutdown_connections 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:53:19.452 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:53:19.454 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.455 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.456 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:53:19.457 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:53:19.458 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:53:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.443+0000 7f787ed5c740 1 -- 172.21.15.28:0/3891055648 wait complete. 2024-09-21T05:53:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f787ed5c740 20 client.16992 trim_cache size 0 max 16384 2024-09-21T05:53:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 msgr2=0x7f785c0b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 0x7f785c0b8c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7870052860 tx=0x7f78700514d0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 msgr2=0x7f78680b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f786c00b450 tx=0x7f786c00b850 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 shutdown_connections 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78680ad480 0x7f78680af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78680acb30 0x7f78680aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f785c0b6730 0x7f785c0b8c40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 --2- 172.21.15.28:0/3633146907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78680a50f0 0x7f78680b3b40 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 >> 172.21.15.28:0/3633146907 conn(0x7f786801cc70 msgr2=0x7f78680b9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 shutdown_connections 2024-09-21T05:53:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:19.444+0000 7f7877586640 1 -- 172.21.15.28:0/3633146907 wait complete. 2024-09-21T05:53:20.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:19 smithi028 ceph-mon[31227]: pgmap v514: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-09-21T05:53:20.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:19 smithi181 ceph-mon[34819]: pgmap v514: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-09-21T05:53:20.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:19 smithi129 ceph-mon[34885]: pgmap v514: 97 pgs: 97 active+clean; 18 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-09-21T05:53:20.409 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:53:20.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:53:20.987 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.672+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3111680580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410ad10 msgr2=0x7fdcf4116fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.672+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3111680580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410ad10 0x7fdcf4116fc0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007290 tx=0x7fdcec032cd0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.674+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3111680580 shutdown_connections 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.674+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3111680580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410ad10 0x7fdcf4116fc0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.674+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3111680580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410a3c0 0x7fdcf410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.674+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3111680580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf410f150 0x7fdcf410f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.674+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3111680580 >> 172.21.15.28:0/3111680580 conn(0x7fdcf40712f0 msgr2=0x7fdcf4073e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:21.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.676+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3111680580 shutdown_connections 2024-09-21T05:53:21.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.680+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3111680580 wait complete. 2024-09-21T05:53:21.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.681+0000 7fdcf8fea640 1 Processor -- start 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 -- start start 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410ad10 0x7fdcf41b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf410f150 0x7fdcf41b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf411a0b0 con 0x7fdcf410a3c0 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf4119f30 con 0x7fdcf410f150 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.685+0000 7fdcf8fea640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf411a230 con 0x7fdcf410ad10 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.686+0000 7fdcf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.686+0000 7fdcf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53198/0 (socket says 172.21.15.28:53198) 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.686+0000 7fdcf259c640 1 -- 172.21.15.28:0/3667294813 learned_addr learned my addr 172.21.15.28:0/3667294813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf1d9b640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410ad10 0x7fdcf41b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 -- 172.21.15.28:0/3667294813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410ad10 msgr2=0x7fdcf41b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410ad10 0x7fdcf41b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 -- 172.21.15.28:0/3667294813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf410f150 msgr2=0x7fdcf41b7270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf410f150 0x7fdcf41b7270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 -- 172.21.15.28:0/3667294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf41b79a0 con 0x7fdcf410a3c0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.687+0000 7fdcf259c640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdce400b4a0 tx=0x7fdce400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.688+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4014070 con 0x7fdcf410a3c0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.688+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdce4010ac0 con 0x7fdcf410a3c0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.688+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4017c20 con 0x7fdcf410a3c0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.688+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf41b7c90 con 0x7fdcf410a3c0 2024-09-21T05:53:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.688+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf40755f0 con 0x7fdcf410a3c0 2024-09-21T05:53:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.694+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fdce4017070 con 0x7fdcf410a3c0 2024-09-21T05:53:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.696+0000 7fdce37fe640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 0x7fdcdc0dacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.696+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fdce40e6990 con 0x7fdcf410a3c0 2024-09-21T05:53:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.697+0000 7fdcf1d9b640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 0x7fdcdc0dacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.699+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf410b840 con 0x7fdcf410a3c0 2024-09-21T05:53:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.704+0000 7fdcf1d9b640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 0x7fdcdc0dacd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdce8005fd0 tx=0x7fdce8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:21.709+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdce40acea0 con 0x7fdcf410a3c0 2024-09-21T05:53:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.170+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcf41125f0 con 0x7fdcf410a3c0 2024-09-21T05:53:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.174+0000 7fdce37fe640 1 -- 172.21.15.28:0/3667294813 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11989 (secure 0 0 0) 0x7fdce40b1280 con 0x7fdcf410a3c0 2024-09-21T05:53:22.176 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:53:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":43,"btime":"2024-09-21T05:53:14:950942+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:53:14.274727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:53:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 43 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 msgr2=0x7fdcdc0dacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 0x7fdcdc0dacd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdce8005fd0 tx=0x7fdce8005850 comp rx=0 tx=0).stop 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 msgr2=0x7fdcf41b1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdce400b4a0 tx=0x7fdce400b960 comp rx=0 tx=0).stop 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 shutdown_connections 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf410f150 0x7fdcf41b7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdcf410ad10 0x7fdcf41b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdcdc0d87c0 0x7fdcdc0dacd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 --2- 172.21.15.28:0/3667294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf410a3c0 0x7fdcf41b1a30 secure :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdce400b4a0 tx=0x7fdce400b960 comp rx=0 tx=0).stop 2024-09-21T05:53:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.182+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 >> 172.21.15.28:0/3667294813 conn(0x7fdcf40712f0 msgr2=0x7fdcf410c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.184+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 shutdown_connections 2024-09-21T05:53:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:22.184+0000 7fdcf8fea640 1 -- 172.21.15.28:0/3667294813 wait complete. 2024-09-21T05:53:22.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:21 smithi028 ceph-mon[31227]: pgmap v515: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 19 MiB/s wr, 4.39k op/s 2024-09-21T05:53:22.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:21 smithi181 ceph-mon[34819]: pgmap v515: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 19 MiB/s wr, 4.39k op/s 2024-09-21T05:53:22.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:21 smithi129 ceph-mon[34885]: pgmap v515: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 19 MiB/s wr, 4.39k op/s 2024-09-21T05:53:23.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:53:23.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:23 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3667294813' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:53:23.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3667294813' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:53:23.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:23 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3667294813' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:53:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:24.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:24 smithi028 ceph-mon[31227]: pgmap v516: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 5.18k op/s 2024-09-21T05:53:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 -- 172.21.15.28:0/2815809682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009c390 msgr2=0x7f82500a2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 --2- 172.21.15.28:0/2815809682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009c390 0x7f82500a2bc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f82580524b0 tx=0x7f8258098170 comp rx=0 tx=0).stop 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 -- 172.21.15.28:0/2815809682 shutdown_connections 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 --2- 172.21.15.28:0/2815809682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009c390 0x7f82500a2bc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 --2- 172.21.15.28:0/2815809682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f825009ba40 0x7f825009be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 --2- 172.21.15.28:0/2815809682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8250096150 0x7f8250096560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 -- 172.21.15.28:0/2815809682 >> 172.21.15.28:0/2815809682 conn(0x7f825000dcb0 msgr2=0x7f825000e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 -- 172.21.15.28:0/2815809682 shutdown_connections 2024-09-21T05:53:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.192+0000 7f825f778640 1 -- 172.21.15.28:0/2815809682 wait complete. 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.193+0000 7f825f778640 1 Processor -- start 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.193+0000 7f825f778640 1 -- start start 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009ba40 0x7f825012cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 0x7f82501321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82500a5fa0 con 0x7f825009ba40 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82500a5e20 con 0x7f825009c390 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825f778640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f82500a6120 con 0x7f8250096150 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825d514640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825d514640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:43972/0 (socket says 172.21.15.28:43972) 2024-09-21T05:53:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825d514640 1 -- 172.21.15.28:0/2115449142 learned_addr learned my addr 172.21.15.28:0/2115449142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825dd15640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 0x7f82501321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.194+0000 7f825cd13640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009ba40 0x7f825012cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 -- 172.21.15.28:0/2115449142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 msgr2=0x7f82501321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 0x7f82501321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 -- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009ba40 msgr2=0x7f825012cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009ba40 0x7f825012cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 -- 172.21.15.28:0/2115449142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82501328e0 con 0x7f8250096150 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825dd15640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 0x7f82501321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825d514640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f825400e9f0 tx=0x7f825400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f824e7fc640 1 -- 172.21.15.28:0/2115449142 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825400cd80 con 0x7f8250096150 2024-09-21T05:53:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f824e7fc640 1 -- 172.21.15.28:0/2115449142 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82540108e0 con 0x7f8250096150 2024-09-21T05:53:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8250132bd0 con 0x7f8250096150 2024-09-21T05:53:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.195+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82500073b0 con 0x7f8250096150 2024-09-21T05:53:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.196+0000 7f824e7fc640 1 -- 172.21.15.28:0/2115449142 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254018980 con 0x7f8250096150 2024-09-21T05:53:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.199+0000 7f824e7fc640 1 -- 172.21.15.28:0/2115449142 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8254010430 con 0x7f8250096150 2024-09-21T05:53:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.199+0000 7f8265cec740 1 Processor -- start 2024-09-21T05:53:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 -- start start 2024-09-21T05:53:24.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380dd7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380dd7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559380d44ea0 con 0x559380a27070 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559380d342d0 con 0x559380a06390 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f8265cec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559380d380e0 con 0x5593809fe1b0 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.200+0000 7f825d514640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:43982/0 (socket says 172.21.15.28:43982) 2024-09-21T05:53:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 -- 172.21.15.28:0/3020049207 learned_addr learned my addr 172.21.15.28:0/3020049207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825dd15640 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380dd7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 -- 172.21.15.28:0/3020049207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 msgr2=0x559380dd7af0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380dd7af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 -- 172.21.15.28:0/3020049207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 msgr2=0x559380dd7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380dd7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 -- 172.21.15.28:0/3020049207 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559380a92650 con 0x5593809fe1b0 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825dd15640 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380dd7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f825d514640 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82540e4f30 tx=0x7f82540ef690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f0c68ebf348d25f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.201+0000 7f82437fe640 1 -- 172.21.15.28:0/3020049207 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540f6070 con 0x5593809fe1b0 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.202+0000 7f82437fe640 1 -- 172.21.15.28:0/3020049207 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82540f2af0 con 0x5593809fe1b0 2024-09-21T05:53:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.202+0000 7f82437fe640 1 -- 172.21.15.28:0/3020049207 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540f19d0 con 0x5593809fe1b0 2024-09-21T05:53:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 -- 172.21.15.28:0/3020049207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 msgr2=0x559380dd75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82540e4f30 tx=0x7f82540ef690 comp rx=0 tx=0).stop 2024-09-21T05:53:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 -- 172.21.15.28:0/3020049207 shutdown_connections 2024-09-21T05:53:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380dd7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5593809fe1b0 0x559380dd75b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 --2- 172.21.15.28:0/3020049207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380dd7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 -- 172.21.15.28:0/3020049207 >> 172.21.15.28:0/3020049207 conn(0x559380a1b460 msgr2=0x559380dd4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f824e7fc640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 0x7f82440b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 -- 172.21.15.28:0/3020049207 shutdown_connections 2024-09-21T05:53:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.203+0000 7f8265cec740 1 -- 172.21.15.28:0/3020049207 wait complete. 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.204+0000 7f825cd13640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 0x7f82440b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.204+0000 7f824e7fc640 1 -- 172.21.15.28:0/2115449142 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f82540e7850 con 0x7f8250096150 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.204+0000 7f8265cec740 1 -- start start 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.204+0000 7f825cd13640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 0x7f82440b8c10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8250005420 tx=0x7f824800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.205+0000 7f8265cec740 1 Processor -- start 2024-09-21T05:53:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.205+0000 7f8265cec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380deebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f825d514640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f825d514640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:45778/0 (socket says 172.21.15.28:45778) 2024-09-21T05:53:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f825cd13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559380b95700 con 0x559380a27070 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559380ccf160 con 0x5593809fe1b0 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f825cd13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:43988/0 (socket says 172.21.15.28:43988) 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559380de5850 con 0x559380a06390 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f825cd13640 1 -- 172.21.15.28:0/3792477096 learned_addr learned my addr 172.21.15.28:0/3792477096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.206+0000 7f8265cec740 10 client.0 _ll_register_callbacks cb 0x559380ce89e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825dd15640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380deebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 -- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 msgr2=0x559380de2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 -- 172.21.15.28:0/3792477096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 msgr2=0x559380deebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380deebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 -- 172.21.15.28:0/3792477096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559380cfcc90 con 0x559380a06390 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825d514640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825dd15640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380deebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:53:24.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f825cd13640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8248010fc0 tx=0x7f8248004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40dc2a5869508e90 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.207+0000 7f8240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824801e830 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82540f2af0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f824801e9d0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f824801ebd0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8265cec740 10 client.26593 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.208+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559380df15e0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.209+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82540f2af0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f824801c3b0 con 0x559380a06390 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 10 client.26593 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 10 client.26593 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 20 client.26593 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809e59f0 0x559380df4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 4 client.26593 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559380d46df0 con 0x5593809e59f0 2024-09-21T05:53:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.210+0000 7f825d514640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809e59f0 0x559380df4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.211+0000 7f825d514640 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809e59f0 0x559380df4ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6f4bca49f4a8480 server_cookie=46c555fdf00d249d in_seq=0 out_seq=0 2024-09-21T05:53:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.211+0000 7f8240ff9640 10 client.26593 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:53:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.212+0000 7f8240ff9640 1 -- 172.21.15.28:0/3792477096 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559380d46df0 con 0x5593809e59f0 2024-09-21T05:53:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.212+0000 7f8240ff9640 10 client.26593 handle_command_reply: tid=0 2024-09-21T05:53:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.212+0000 7f8265cec740 1 client.26593 shutdown 2024-09-21T05:53:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.213+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 msgr2=0x559380de2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.213+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8248010fc0 tx=0x7f8248004730 comp rx=0 tx=0).stop 2024-09-21T05:53:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 shutdown_connections 2024-09-21T05:53:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809e59f0 0x559380df4ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559380a27070 0x559380deebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de2c70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 --2- 172.21.15.28:0/3792477096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593809fe1b0 0x559380de2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 >> 172.21.15.28:0/3792477096 conn(0x559380a1b460 msgr2=0x559380dda200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8240ff9640 0 client.26593 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:53:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 shutdown_connections 2024-09-21T05:53:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 1 -- 172.21.15.28:0/3792477096 wait complete. 2024-09-21T05:53:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.214+0000 7f8265cec740 20 client.26593 trim_cache size 0 max 16384 2024-09-21T05:53:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 Processor -- start 2024-09-21T05:53:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 -- start start 2024-09-21T05:53:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380de24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 0x559380de2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00e7a0 con 0x5593809fe1b0 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f823c010340 con 0x559380a06390 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f8265cec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00ee70 con 0x559380a27070 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f825d514640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f825d514640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53238/0 (socket says 172.21.15.28:53238) 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f825d514640 1 -- 172.21.15.28:0/3555106406 learned_addr learned my addr 172.21.15.28:0/3555106406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f825dd15640 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 0x559380de2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.238+0000 7f825cd13640 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380de24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.239+0000 7f825dd15640 1 -- 172.21.15.28:0/3555106406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 msgr2=0x559380de24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.239+0000 7f825dd15640 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380de24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.239+0000 7f825dd15640 1 -- 172.21.15.28:0/3555106406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 msgr2=0x559380deead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.239+0000 7f825dd15640 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.239+0000 7f825dd15640 1 -- 172.21.15.28:0/3555106406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559380dd9ed0 con 0x559380a27070 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.240+0000 7f825dd15640 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 0x559380de2a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8258052910 tx=0x7f825809a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4d19a7e06dc7b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.240+0000 7f8242ffd640 1 -- 172.21.15.28:0/3555106406 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82580b3070 con 0x559380a27070 2024-09-21T05:53:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.240+0000 7f8242ffd640 1 -- 172.21.15.28:0/3555106406 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f825809c040 con 0x559380a27070 2024-09-21T05:53:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.240+0000 7f8242ffd640 1 -- 172.21.15.28:0/3555106406 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825806a960 con 0x559380a27070 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 -- 172.21.15.28:0/3555106406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 msgr2=0x559380de2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 0x559380de2a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8258052910 tx=0x7f825809a040 comp rx=0 tx=0).stop 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 -- 172.21.15.28:0/3555106406 shutdown_connections 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a27070 0x559380de2a20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a06390 0x559380de24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 --2- 172.21.15.28:0/3555106406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 -- 172.21.15.28:0/3555106406 >> 172.21.15.28:0/3555106406 conn(0x559380a1b460 msgr2=0x559380de0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 -- 172.21.15.28:0/3555106406 shutdown_connections 2024-09-21T05:53:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.242+0000 7f8265cec740 1 -- 172.21.15.28:0/3555106406 wait complete. 2024-09-21T05:53:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.243+0000 7f8265cec740 1 -- start start 2024-09-21T05:53:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.243+0000 7f8265cec740 1 Processor -- start 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a27070 0x559380de69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00fc70 con 0x5593809fe1b0 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00f540 con 0x559380a27070 2024-09-21T05:53:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00c780 con 0x559380a06390 2024-09-21T05:53:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.244+0000 7f8265cec740 10 client.0 _ll_register_callbacks cb 0x7f824800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.246+0000 7f825cd13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.246+0000 7f825cd13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44010/0 (socket says 172.21.15.28:44010) 2024-09-21T05:53:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.246+0000 7f825cd13640 1 -- 172.21.15.28:0/2376512491 learned_addr learned my addr 172.21.15.28:0/2376512491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.246+0000 7f825d514640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825dd15640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a27070 0x559380de69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825cd13640 1 -- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a27070 msgr2=0x559380de69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825cd13640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a27070 0x559380de69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825cd13640 1 -- 172.21.15.28:0/2376512491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 msgr2=0x559380deafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825cd13640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825cd13640 1 -- 172.21.15.28:0/2376512491 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559380de6ef0 con 0x559380a06390 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.247+0000 7f825d514640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.248+0000 7f825cd13640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8248011010 tx=0x7f824801e8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f27a50c92eb5fff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.248+0000 7f8240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.248+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824803d030 con 0x559380a06390 2024-09-21T05:53:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.248+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f824802f620 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.248+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825809c040 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.249+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f8248048050 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.249+0000 7f8265cec740 10 client.26599 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.249+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559380e5f0c0 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.249+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f825809c040 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f8248026400 con 0x559380a06390 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 10 client.26599 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 10 client.26599 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 20 client.26599 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809bada0 0x559380e618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 4 client.26599 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:53:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559380d428a0 con 0x5593809bada0 2024-09-21T05:53:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.250+0000 7f825d514640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809bada0 0x559380e618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.252+0000 7f825d514640 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809bada0 0x559380e618d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9d03507a155eca3 server_cookie=28774320b9d70b3 in_seq=0 out_seq=0 2024-09-21T05:53:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.252+0000 7f8240ff9640 10 client.26599 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8240ff9640 1 -- 172.21.15.28:0/2376512491 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559380d428a0 con 0x5593809bada0 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8240ff9640 10 client.26599 handle_command_reply: tid=0 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 client.26599 shutdown 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 msgr2=0x559380de6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8248011010 tx=0x7f824801e8e0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 shutdown_connections 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5593809bada0 0x559380e618d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559380a27070 0x559380de69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559380a06390 0x559380de6470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8240ff9640 0 client.26599 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 --2- 172.21.15.28:0/2376512491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593809fe1b0 0x559380deafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.253+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 >> 172.21.15.28:0/2376512491 conn(0x559380a1b460 msgr2=0x559380de0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 shutdown_connections 2024-09-21T05:53:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f8265cec740 1 -- 172.21.15.28:0/2376512491 wait complete. 2024-09-21T05:53:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f8265cec740 20 client.26599 trim_cache size 0 max 16384 2024-09-21T05:53:24.258 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 msgr2=0x7f82440b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 0x7f82440b8c10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8250005420 tx=0x7f824800b390 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 msgr2=0x7f825012c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.254+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f825400e9f0 tx=0x7f825400eeb0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 shutdown_connections 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825009c390 0x7f82501321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f82440b6700 0x7f82440b8c10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825009ba40 0x7f825012cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 --2- 172.21.15.28:0/2115449142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8250096150 0x7f825012c970 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 >> 172.21.15.28:0/2115449142 conn(0x7f825000dcb0 msgr2=0x7f8250099a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 shutdown_connections 2024-09-21T05:53:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:24.255+0000 7f825f778640 1 -- 172.21.15.28:0/2115449142 wait complete. 2024-09-21T05:53:24.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:24 smithi181 ceph-mon[34819]: pgmap v516: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 5.18k op/s 2024-09-21T05:53:24.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:24 smithi129 ceph-mon[34885]: pgmap v516: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 5.18k op/s 2024-09-21T05:53:25.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:53:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:26.294 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:26 smithi181 ceph-mon[34819]: pgmap v517: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 5.79k op/s 2024-09-21T05:53:26.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:26 smithi028 ceph-mon[31227]: pgmap v517: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 5.79k op/s 2024-09-21T05:53:26.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.313+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/2923277528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4075dd0 msgr2=0x7f3eb40761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.313+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/2923277528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4075dd0 0x7f3eb40761e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009900 tx=0x7f3ea802a440 comp rx=0 tx=0).stop 2024-09-21T05:53:26.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/2923277528 shutdown_connections 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/2923277528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3eb4114470 0x7f3eb4116940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/2923277528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4075dd0 0x7f3eb40761e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/2923277528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4077380 0x7f3eb40757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/2923277528 >> 172.21.15.28:0/2923277528 conn(0x7f3eb40ff630 msgr2=0x7f3eb4101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/2923277528 shutdown_connections 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.316+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/2923277528 wait complete. 2024-09-21T05:53:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.318+0000 7f3eb8bc0640 1 Processor -- start 2024-09-21T05:53:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.318+0000 7f3eb8bc0640 1 -- start start 2024-09-21T05:53:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.318+0000 7f3eb8bc0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 0x7f3eb4074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb8bc0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 0x7f3eb4070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb8bc0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3eb4114470 0x7f3eb40706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb8bc0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb406e510 con 0x7f3eb4077380 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb8bc0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb406e390 con 0x7f3eb4114470 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb8bc0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb406e690 con 0x7f3eb4075dd0 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 0x7f3eb4074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 0x7f3eb4074e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51896/0 (socket says 172.21.15.28:51896) 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb259c640 1 -- 172.21.15.28:0/162909054 learned_addr learned my addr 172.21.15.28:0/162909054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.319+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 0x7f3eb4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.321+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/162909054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 msgr2=0x7f3eb4074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.321+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 0x7f3eb4074e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.321+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/162909054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3eb4114470 msgr2=0x7f3eb40706a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.321+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3eb4114470 0x7f3eb40706a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.321+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/162909054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb4070e20 con 0x7f3eb4077380 2024-09-21T05:53:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.322+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 0x7f3eb4070160 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009850 tx=0x7f3ea8003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.324+0000 7f3ea37fe640 1 -- 172.21.15.28:0/162909054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea803f070 con 0x7f3eb4077380 2024-09-21T05:53:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.325+0000 7f3ea37fe640 1 -- 172.21.15.28:0/162909054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea8003e40 con 0x7f3eb4077380 2024-09-21T05:53:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.325+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb41b5790 con 0x7f3eb4077380 2024-09-21T05:53:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.325+0000 7f3ea37fe640 1 -- 172.21.15.28:0/162909054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8035030 con 0x7f3eb4077380 2024-09-21T05:53:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.325+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb41b5b80 con 0x7f3eb4077380 2024-09-21T05:53:26.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.331+0000 7f3ea37fe640 1 -- 172.21.15.28:0/162909054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3ea8042070 con 0x7f3eb4077380 2024-09-21T05:53:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.332+0000 7f3ea37fe640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 0x7f3e9c0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.334+0000 7f3eb259c640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 0x7f3e9c0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.334+0000 7f3ea37fe640 1 -- 172.21.15.28:0/162909054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3ea8108e00 con 0x7f3eb4077380 2024-09-21T05:53:26.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.335+0000 7f3ebf134740 1 Processor -- start 2024-09-21T05:53:26.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.336+0000 7f3eb259c640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 0x7f3e9c0b8d10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4074be0 tx=0x7f3ea4009190 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.336+0000 7f3ebf134740 1 -- start start 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9881b0 0x56513ed58800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed58d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56513ecceea0 con 0x56513e9b1070 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56513ecbe2d0 con 0x56513e9881b0 2024-09-21T05:53:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.337+0000 7f3ebf134740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56513ecc20e0 con 0x56513e990390 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.338+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.338+0000 7f3eb2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed58d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.338+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40042/0 (socket says 172.21.15.28:40042) 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.338+0000 7f3eb259c640 1 -- 172.21.15.28:0/201385373 learned_addr learned my addr 172.21.15.28:0/201385373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.338+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9881b0 0x56513ed58800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 -- 172.21.15.28:0/201385373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 msgr2=0x56513ed58d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed58d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 -- 172.21.15.28:0/201385373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9881b0 msgr2=0x56513ed58800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9881b0 0x56513ed58800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 -- 172.21.15.28:0/201385373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513ea1c650 con 0x56513e9b1070 2024-09-21T05:53:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.339+0000 7f3eb259c640 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400ec20 tx=0x7f3ea4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6beb742eafc398 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.340+0000 7f3ea0ff9640 1 -- 172.21.15.28:0/201385373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea401b040 con 0x56513e9b1070 2024-09-21T05:53:26.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.340+0000 7f3ea0ff9640 1 -- 172.21.15.28:0/201385373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea4024070 con 0x56513e9b1070 2024-09-21T05:53:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.340+0000 7f3ea0ff9640 1 -- 172.21.15.28:0/201385373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea401f3f0 con 0x56513e9b1070 2024-09-21T05:53:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.342+0000 7f3ebf134740 1 -- 172.21.15.28:0/201385373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 msgr2=0x56513ed582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.342+0000 7f3ebf134740 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400ec20 tx=0x7f3ea4011000 comp rx=0 tx=0).stop 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.343+0000 7f3ebf134740 1 -- 172.21.15.28:0/201385373 shutdown_connections 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.343+0000 7f3ebf134740 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed58d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.343+0000 7f3ebf134740 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9881b0 0x56513ed58800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.343+0000 7f3ebf134740 1 --2- 172.21.15.28:0/201385373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9b1070 0x56513ed582c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.343+0000 7f3ebf134740 1 -- 172.21.15.28:0/201385373 >> 172.21.15.28:0/201385373 conn(0x56513e9a5460 msgr2=0x56513ed561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.345+0000 7f3ebf134740 1 -- 172.21.15.28:0/201385373 shutdown_connections 2024-09-21T05:53:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.346+0000 7f3ebf134740 1 -- 172.21.15.28:0/201385373 wait complete. 2024-09-21T05:53:26.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.346+0000 7f3ebf134740 1 -- start start 2024-09-21T05:53:26.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.346+0000 7f3ebf134740 1 Processor -- start 2024-09-21T05:53:26.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9881b0 0x56513ed63a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e990390 0x56513ed63f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56513eb1f700 con 0x56513e990390 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56513ec59160 con 0x56513e9b1070 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56513ed66aa0 con 0x56513e9881b0 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3ebf134740 10 client.0 _ll_register_callbacks cb 0x56513ec729e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.347+0000 7f3eb2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e990390 0x56513ed63f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36752/0 (socket says 172.21.15.28:36752) 2024-09-21T05:53:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 -- 172.21.15.28:0/3974098991 learned_addr learned my addr 172.21.15.28:0/3974098991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 -- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9881b0 msgr2=0x56513ed63a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9881b0 0x56513ed63a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 -- 172.21.15.28:0/3974098991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e990390 msgr2=0x56513ed63f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e990390 0x56513ed63f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.348+0000 7f3eb2d9d640 1 -- 172.21.15.28:0/3974098991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56513ec86c90 con 0x56513e9b1070 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.349+0000 7f3eb2d9d640 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e710 tx=0x7f3eac00eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2913e13c0c017cc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.349+0000 7f3e99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.349+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac004d20 con 0x56513e9b1070 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.349+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea4024070 con 0x56513e9b1070 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.350+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3eac00c8b0 con 0x56513e9b1070 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.350+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3eac00cad0 con 0x56513e9b1070 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.350+0000 7f3ebf134740 10 client.26600 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:26.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.350+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56513ed72920 con 0x56513e9b1070 2024-09-21T05:53:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.351+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eac00cc70 con 0x56513e9b1070 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f3eac010430 con 0x56513e9b1070 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 10 client.26600 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 10 client.26600 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 20 client.26600 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e96f9f0 0x56513ed75f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 4 client.26600 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.353+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3ea4024070 con 0x56513e96f9f0 2024-09-21T05:53:26.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.354+0000 7f3eb259c640 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e96f9f0 0x56513ed75f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.355+0000 7f3eb259c640 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e96f9f0 0x56513ed75f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce051a29a9e0304f server_cookie=378b08123352891f in_seq=0 out_seq=0 2024-09-21T05:53:26.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.355+0000 7f3e99ffb640 10 client.26600 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:53:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.356+0000 7f3e99ffb640 1 -- 172.21.15.28:0/3974098991 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f3ea4024070 con 0x56513e96f9f0 2024-09-21T05:53:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.356+0000 7f3e99ffb640 10 client.26600 handle_command_reply: tid=0 2024-09-21T05:53:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.356+0000 7f3ebf134740 1 client.26600 shutdown 2024-09-21T05:53:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 msgr2=0x56513ed6fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e710 tx=0x7f3eac00eb10 comp rx=0 tx=0).stop 2024-09-21T05:53:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 shutdown_connections 2024-09-21T05:53:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e96f9f0 0x56513ed75f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed6fdf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e990390 0x56513ed63f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 --2- 172.21.15.28:0/3974098991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9881b0 0x56513ed63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 >> 172.21.15.28:0/3974098991 conn(0x56513e9a5460 msgr2=0x56513ed5b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.357+0000 7f3e99ffb640 0 client.26600 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.358+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 shutdown_connections 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.358+0000 7f3ebf134740 1 -- 172.21.15.28:0/3974098991 wait complete. 2024-09-21T05:53:26.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.358+0000 7f3ebf134740 20 client.26600 trim_cache size 0 max 16384 2024-09-21T05:53:26.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:26 smithi129 ceph-mon[34885]: pgmap v517: 97 pgs: 97 active+clean; 18 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 5.79k op/s 2024-09-21T05:53:26.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 Processor -- start 2024-09-21T05:53:26.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 -- start start 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed6fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed60c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9400e660 con 0x56513e9881b0 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e94010200 con 0x56513e9b1070 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.384+0000 7f3ebf134740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9400ed30 con 0x56513e990390 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.385+0000 7f3eb2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed60c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.385+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.385+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40052/0 (socket says 172.21.15.28:40052) 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.385+0000 7f3eb259c640 1 -- 172.21.15.28:0/944519785 learned_addr learned my addr 172.21.15.28:0/944519785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed6fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 -- 172.21.15.28:0/944519785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 msgr2=0x56513ed6fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed6fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 -- 172.21.15.28:0/944519785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 msgr2=0x56513ed60c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed60c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 -- 172.21.15.28:0/944519785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513ed5e670 con 0x56513e9881b0 2024-09-21T05:53:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.386+0000 7f3eb259c640 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4017010 tx=0x7f3ea4013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7b336ae6ee5573 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.387+0000 7f3e9bfff640 1 -- 172.21.15.28:0/944519785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea402f070 con 0x56513e9881b0 2024-09-21T05:53:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.387+0000 7f3e9bfff640 1 -- 172.21.15.28:0/944519785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea4024040 con 0x56513e9881b0 2024-09-21T05:53:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.388+0000 7f3e9bfff640 1 -- 172.21.15.28:0/944519785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4013870 con 0x56513e9881b0 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 -- 172.21.15.28:0/944519785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 msgr2=0x56513ed63ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4017010 tx=0x7f3ea4013b10 comp rx=0 tx=0).stop 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 -- 172.21.15.28:0/944519785 shutdown_connections 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e9b1070 0x56513ed60c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e990390 0x56513ed6fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 --2- 172.21.15.28:0/944519785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed63ed0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.389+0000 7f3ebf134740 1 -- 172.21.15.28:0/944519785 >> 172.21.15.28:0/944519785 conn(0x56513e9a5460 msgr2=0x56513ed637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.390+0000 7f3ebf134740 1 -- 172.21.15.28:0/944519785 shutdown_connections 2024-09-21T05:53:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.390+0000 7f3ebf134740 1 -- 172.21.15.28:0/944519785 wait complete. 2024-09-21T05:53:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.390+0000 7f3ebf134740 1 -- start start 2024-09-21T05:53:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.390+0000 7f3ebf134740 1 Processor -- start 2024-09-21T05:53:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed6c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9b1070 0x56513ed67ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9400fb30 con 0x56513e9881b0 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9400f400 con 0x56513e990390 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9400c6f0 con 0x56513e9b1070 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3ebf134740 10 client.0 _ll_register_callbacks cb 0x7f3eac008430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3eb1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3eb1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36760/0 (socket says 172.21.15.28:36760) 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/2047191596 learned_addr learned my addr 172.21.15.28:0/2047191596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.391+0000 7f3eb259c640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed6c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.392+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9b1070 msgr2=0x56513ed67ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.392+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9b1070 0x56513ed67ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.392+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/2047191596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 msgr2=0x56513ed6c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.392+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed6c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.392+0000 7f3eb1d9b640 1 -- 172.21.15.28:0/2047191596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56513ed68660 con 0x56513e990390 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.393+0000 7f3eb1d9b640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80cbee0 tx=0x7f3ea80ce6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbedc73effb4a9c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.393+0000 7f3e99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.393+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8049070 con 0x56513e990390 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.393+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56513ed5f5c0 con 0x56513e990390 2024-09-21T05:53:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.393+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f3ea80d8200 con 0x56513e990390 2024-09-21T05:53:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.394+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3ea80cf370 con 0x56513e990390 2024-09-21T05:53:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.394+0000 7f3ebf134740 10 client.26606 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.394+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56513edf1860 con 0x56513e990390 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56513ed5f5c0 con 0x56513e990390 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f3ea80ca5c0 con 0x56513e990390 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 10 client.26606 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 10 client.26606 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 20 client.26606 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e944da0 0x56513edf4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 4 client.26606 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.396+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56513ecd0df0 con 0x56513e944da0 2024-09-21T05:53:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.397+0000 7f3eb259c640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e944da0 0x56513edf4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.397+0000 7f3e99ffb640 10 client.26606 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:53:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3eb259c640 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e944da0 0x56513edf4070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=705d2efaa9fb7f69 server_cookie=3e2f6c5b3e28e809 in_seq=0 out_seq=0 2024-09-21T05:53:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3e99ffb640 1 -- 172.21.15.28:0/2047191596 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56513ecd0df0 con 0x56513e944da0 2024-09-21T05:53:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3e99ffb640 10 client.26606 handle_command_reply: tid=0 2024-09-21T05:53:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3ebf134740 1 client.26606 shutdown 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 msgr2=0x56513ed679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.398+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80cbee0 tx=0x7f3ea80ce6e0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 shutdown_connections 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56513e944da0 0x56513edf4070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56513e9b1070 0x56513ed67ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56513e990390 0x56513ed679b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 --2- 172.21.15.28:0/2047191596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56513e9881b0 0x56513ed6c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 >> 172.21.15.28:0/2047191596 conn(0x56513e9a5460 msgr2=0x56513ed64ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3e99ffb640 0 client.26606 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 shutdown_connections 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 1 -- 172.21.15.28:0/2047191596 wait complete. 2024-09-21T05:53:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.399+0000 7f3ebf134740 20 client.26606 trim_cache size 0 max 16384 2024-09-21T05:53:26.403 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 msgr2=0x7f3e9c0b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 0x7f3e9c0b8d10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4074be0 tx=0x7f3ea4009190 comp rx=0 tx=0).stop 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 msgr2=0x7f3eb4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 0x7f3eb4070160 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009850 tx=0x7f3ea8003cd0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 shutdown_connections 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3eb4114470 0x7f3eb40706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3e9c0b6800 0x7f3e9c0b8d10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb4077380 0x7f3eb4070160 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 --2- 172.21.15.28:0/162909054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3eb4075dd0 0x7f3eb4074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 >> 172.21.15.28:0/162909054 conn(0x7f3eb40ff630 msgr2=0x7f3eb4110370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 shutdown_connections 2024-09-21T05:53:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:26.401+0000 7f3eb8bc0640 1 -- 172.21.15.28:0/162909054 wait complete. 2024-09-21T05:53:27.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:53:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.175+0000 7fd6716d5640 1 -- 172.21.15.28:0/1870157468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c104650 msgr2=0x7fd66c104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.175+0000 7fd6716d5640 1 --2- 172.21.15.28:0/1870157468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c104650 0x7fd66c104a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009860 tx=0x7fd65c02a340 comp rx=0 tx=0).stop 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.175+0000 7fd6716d5640 1 -- 172.21.15.28:0/1870157468 shutdown_connections 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.176+0000 7fd6716d5640 1 --2- 172.21.15.28:0/1870157468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c10dc00 0x7fd66c110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.176+0000 7fd6716d5640 1 --2- 172.21.15.28:0/1870157468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c10d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.176+0000 7fd6716d5640 1 --2- 172.21.15.28:0/1870157468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c104650 0x7fd66c104a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.176+0000 7fd6716d5640 1 -- 172.21.15.28:0/1870157468 >> 172.21.15.28:0/1870157468 conn(0x7fd66c0fe2c0 msgr2=0x7fd66c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.177+0000 7fd6716d5640 1 -- 172.21.15.28:0/1870157468 shutdown_connections 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.177+0000 7fd6716d5640 1 -- 172.21.15.28:0/1870157468 wait complete. 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.178+0000 7fd6716d5640 1 Processor -- start 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.179+0000 7fd6716d5640 1 -- start start 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.179+0000 7fd6716d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c104650 0x7fd66c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.179+0000 7fd6716d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd6716d5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 0x7fd66c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd6716d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1138d0 con 0x7fd66c105080 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd6716d5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c113750 con 0x7fd66c104650 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd6716d5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c113a50 con 0x7fd66c10dc00 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c19a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40070/0 (socket says 172.21.15.28:40070) 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.180+0000 7fd66b7fe640 1 -- 172.21.15.28:0/4025136593 learned_addr learned my addr 172.21.15.28:0/4025136593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.181+0000 7fd66bfff640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c104650 0x7fd66c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.181+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 0x7fd66c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 -- 172.21.15.28:0/4025136593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c104650 msgr2=0x7fd66c19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c104650 0x7fd66c19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 -- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 msgr2=0x7fd66c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 -- 172.21.15.28:0/4025136593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66c1a02b0 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.182+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 0x7fd66c19fb80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd664009fd0 tx=0x7fd66400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.183+0000 7fd6697fa640 1 -- 172.21.15.28:0/4025136593 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66400ce20 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.183+0000 7fd6697fa640 1 -- 172.21.15.28:0/4025136593 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd664005020 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.183+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66c1a05a0 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.184+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66c1a6d00 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.184+0000 7fd6697fa640 1 -- 172.21.15.28:0/4025136593 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664010690 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.188+0000 7fd6697fa640 1 -- 172.21.15.28:0/4025136593 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd664010910 con 0x7fd66c10dc00 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.188+0000 7fd678eab740 1 Processor -- start 2024-09-21T05:53:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.188+0000 7fd678eab740 1 -- start start 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a05d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a05dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a05e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558259fd3ea0 con 0x558259cb6070 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558259fc32d0 con 0x558259c8d1b0 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.189+0000 7fd678eab740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558259fc70e0 con 0x558259c95390 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd66bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a05dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a05e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a05d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a05d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40074/0 (socket says 172.21.15.28:40074) 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1935178044 learned_addr learned my addr 172.21.15.28:0/1935178044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd6697fa640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 0x7fd6540b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.190+0000 7fd6697fa640 1 -- 172.21.15.28:0/4025136593 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd6640e8dc0 con 0x7fd66c10dc00 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1935178044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 msgr2=0x55825a05dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a05dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1935178044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 msgr2=0x55825a05d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a05d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1935178044 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558259d21650 con 0x558259c95390 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.191+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 0x7fd6540b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.192+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a05e2e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd660012610 tx=0x7fd660009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdcbc35ac1e16cf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.192+0000 7fd670ed4640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 0x7fd6540b8d90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd6640abe40 tx=0x7fd66401f000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.192+0000 7fd6527fc640 1 -- 172.21.15.28:0/1935178044 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660004030 con 0x558259c95390 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.192+0000 7fd6527fc640 1 -- 172.21.15.28:0/1935178044 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6600054d0 con 0x558259c95390 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.192+0000 7fd6527fc640 1 -- 172.21.15.28:0/1935178044 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660004a90 con 0x558259c95390 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.193+0000 7fd678eab740 1 -- 172.21.15.28:0/1935178044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 msgr2=0x55825a05e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.193+0000 7fd678eab740 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a05e2e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd660012610 tx=0x7fd660009460 comp rx=0 tx=0).stop 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 -- 172.21.15.28:0/1935178044 shutdown_connections 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a05e2e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a05dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 --2- 172.21.15.28:0/1935178044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a05d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 -- 172.21.15.28:0/1935178044 >> 172.21.15.28:0/1935178044 conn(0x558259caa460 msgr2=0x55825a05b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.194+0000 7fd678eab740 1 -- 172.21.15.28:0/1935178044 shutdown_connections 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.195+0000 7fd678eab740 1 -- 172.21.15.28:0/1935178044 wait complete. 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.195+0000 7fd678eab740 1 -- start start 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 Processor -- start 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a06fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a069000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558259e24700 con 0x558259cb6070 2024-09-21T05:53:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558259f5e160 con 0x558259c8d1b0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55825a070370 con 0x558259c95390 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.196+0000 7fd678eab740 10 client.0 _ll_register_callbacks cb 0x558259f779e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40084/0 (socket says 172.21.15.28:40084) 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 -- 172.21.15.28:0/2377896529 learned_addr learned my addr 172.21.15.28:0/2377896529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a06fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 -- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 msgr2=0x55825a069000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a069000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 -- 172.21.15.28:0/2377896529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 msgr2=0x55825a06fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a06fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.197+0000 7fd66b7fe640 1 -- 172.21.15.28:0/2377896529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558259f8bc90 con 0x558259cb6070 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd6600057c0 tx=0x7fd6600044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954ed9a3022bb6d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd64b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660028040 con 0x558259cb6070 2024-09-21T05:53:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd660004a10 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6600054d0 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.198+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd660004c10 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.199+0000 7fd678eab740 10 client.17043 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.199+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55825a080560 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.199+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6600054d0 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.199+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fd66001c020 con 0x558259cb6070 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 10 client.17043 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 20 client.17043 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x558259c749f0 0x55825a0838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 4 client.17043 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.200+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558259fd5df0 con 0x558259c749f0 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.201+0000 7fd66bfff640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x558259c749f0 0x55825a0838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.202+0000 7fd66bfff640 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x558259c749f0 0x55825a0838c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed72d2cd6513dd9a server_cookie=5644b64399229919 in_seq=0 out_seq=0 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.202+0000 7fd64b7fe640 10 client.17043 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.203+0000 7fd64b7fe640 1 -- 172.21.15.28:0/2377896529 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558259fd5df0 con 0x558259c749f0 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.203+0000 7fd64b7fe640 10 client.17043 handle_command_reply: tid=0 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.203+0000 7fd678eab740 1 client.17043 shutdown 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.203+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 msgr2=0x55825a069710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.203+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd6600057c0 tx=0x7fd6600044f0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 shutdown_connections 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x558259c749f0 0x55825a0838c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259cb6070 0x55825a069710 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c95390 0x55825a069000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 --2- 172.21.15.28:0/2377896529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c8d1b0 0x55825a06fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 >> 172.21.15.28:0/2377896529 conn(0x558259caa460 msgr2=0x55825a0609b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd64b7fe640 0 client.17043 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 shutdown_connections 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 1 -- 172.21.15.28:0/2377896529 wait complete. 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.204+0000 7fd678eab740 20 client.17043 trim_cache size 0 max 16384 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.228+0000 7fd678eab740 1 Processor -- start 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.228+0000 7fd678eab740 1 -- start start 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c95390 0x55825a06cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259cb6070 0x55825a06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ad2c0 con 0x558259c95390 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1aee60 con 0x558259cb6070 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd678eab740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ad990 con 0x558259c8d1b0 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:51990/0 (socket says 172.21.15.28:51990) 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd66b7fe640 1 -- 172.21.15.28:0/3500049664 learned_addr learned my addr 172.21.15.28:0/3500049664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.229+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259cb6070 0x55825a06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 -- 172.21.15.28:0/3500049664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259cb6070 msgr2=0x55825a06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259cb6070 0x55825a06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 -- 172.21.15.28:0/3500049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c95390 msgr2=0x55825a06cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c95390 0x55825a06cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 -- 172.21.15.28:0/3500049664 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825a06ea40 con 0x558259c8d1b0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6600096c0 tx=0x7fd660012320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb10ad1a9b6aae79 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd651ffb640 1 -- 172.21.15.28:0/3500049664 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66001e480 con 0x558259c8d1b0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd651ffb640 1 -- 172.21.15.28:0/3500049664 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd660009910 con 0x558259c8d1b0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.230+0000 7fd651ffb640 1 -- 172.21.15.28:0/3500049664 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66000d860 con 0x558259c8d1b0 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- 172.21.15.28:0/3500049664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 msgr2=0x55825a06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6600096c0 tx=0x7fd660012320 comp rx=0 tx=0).stop 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- 172.21.15.28:0/3500049664 shutdown_connections 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259cb6070 0x55825a06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c95390 0x55825a06cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 --2- 172.21.15.28:0/3500049664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259c8d1b0 0x55825a06c570 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6600096c0 tx=0x7fd660012320 comp rx=0 tx=0).stop 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- 172.21.15.28:0/3500049664 >> 172.21.15.28:0/3500049664 conn(0x558259caa460 msgr2=0x55825a0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- 172.21.15.28:0/3500049664 shutdown_connections 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- 172.21.15.28:0/3500049664 wait complete. 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.232+0000 7fd678eab740 1 -- start start 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 Processor -- start 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259cb6070 0x55825a07a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c49da0 0x55825a07ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ae790 con 0x558259c49da0 2024-09-21T05:53:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ae060 con 0x558259c95390 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ab2a0 con 0x558259cb6070 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.233+0000 7fd678eab740 10 client.0 _ll_register_callbacks cb 0x7fd660005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.234+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.234+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36836/0 (socket says 172.21.15.28:36836) 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.234+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1618513865 learned_addr learned my addr 172.21.15.28:0/1618513865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.234+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c49da0 0x55825a07ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.234+0000 7fd66bfff640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259cb6070 0x55825a07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259cb6070 msgr2=0x55825a07a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259cb6070 0x55825a07a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1618513865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c49da0 msgr2=0x55825a07ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c49da0 0x55825a07ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 -- 172.21.15.28:0/1618513865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55825a07b270 con 0x558259c95390 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd66000d410 tx=0x7fd66000d500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23b9d4a46b880b59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.235+0000 7fd64b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66001e480 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd660009910 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd66000d5e0 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd6600093c0 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd678eab740 10 client.26624 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.236+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55825a0ee0b0 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd660009910 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fd66001c070 con 0x558259c95390 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 10 client.26624 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 10 client.26624 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 20 client.26624 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55825a0ee580 0x55825a0f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 4 client.26624 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558259fd18a0 con 0x55825a0ee580 2024-09-21T05:53:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.237+0000 7fd66bfff640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55825a0ee580 0x55825a0f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd66bfff640 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55825a0ee580 0x55825a0f09a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=422c3830c18a59fe server_cookie=279306582cbf8954 in_seq=0 out_seq=0 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd64b7fe640 10 client.26624 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd64b7fe640 1 -- 172.21.15.28:0/1618513865 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558259fd18a0 con 0x55825a0ee580 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd64b7fe640 10 client.26624 handle_command_reply: tid=0 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd678eab740 1 client.26624 shutdown 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 msgr2=0x55825a07e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.238+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd66000d410 tx=0x7fd66000d500 comp rx=0 tx=0).stop 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 shutdown_connections 2024-09-21T05:53:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55825a0ee580 0x55825a0f09a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd64b7fe640 0 client.26624 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558259c49da0 0x55825a07ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558259cb6070 0x55825a07a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 --2- 172.21.15.28:0/1618513865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558259c95390 0x55825a07e870 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 >> 172.21.15.28:0/1618513865 conn(0x558259caa460 msgr2=0x55825a05e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 shutdown_connections 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 1 -- 172.21.15.28:0/1618513865 wait complete. 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.239+0000 7fd678eab740 20 client.26624 trim_cache size 0 max 16384 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.240+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 msgr2=0x7fd6540b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.240+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 0x7fd6540b8d90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd6640abe40 tx=0x7fd66401f000 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 msgr2=0x7fd66c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 0x7fd66c19fb80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd664009fd0 tx=0x7fd66400ef10 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 shutdown_connections 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd66c10dc00 0x7fd66c19fb80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd6540b6880 0x7fd6540b8d90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c105080 0x7fd66c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 --2- 172.21.15.28:0/4025136593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66c104650 0x7fd66c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 >> 172.21.15.28:0/4025136593 conn(0x7fd66c0fe2c0 msgr2=0x7fd66c1077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 shutdown_connections 2024-09-21T05:53:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:28.241+0000 7fd6716d5640 1 -- 172.21.15.28:0/4025136593 wait complete. 2024-09-21T05:53:28.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:28 smithi181 ceph-mon[34819]: pgmap v518: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 7.42k op/s 2024-09-21T05:53:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:28 smithi028 ceph-mon[31227]: pgmap v518: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 7.42k op/s 2024-09-21T05:53:28.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:28 smithi129 ceph-mon[34885]: pgmap v518: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 7.42k op/s 2024-09-21T05:53:29.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.726+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/235785060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 msgr2=0x7fa6e010d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.726+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/235785060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e010d790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009860 tx=0x7fa6d002a340 comp rx=0 tx=0).stop 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.727+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/235785060 shutdown_connections 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.727+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/235785060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 0x7fa6e0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.727+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/235785060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e010d790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.727+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/235785060 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e0104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.727+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/235785060 >> 172.21.15.28:0/235785060 conn(0x7fa6e007a740 msgr2=0x7fa6e007ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.728+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/235785060 shutdown_connections 2024-09-21T05:53:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.728+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/235785060 wait complete. 2024-09-21T05:53:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.729+0000 7fa6e7a7e640 1 Processor -- start 2024-09-21T05:53:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.729+0000 7fa6e7a7e640 1 -- start start 2024-09-21T05:53:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e01a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 0x7fa6e01a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01117d0 con 0x7fa6e010dd60 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e0111650 con 0x7fa6e0105250 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e7a7e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e0111950 con 0x7fa6e0104900 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:52014/0 (socket says 172.21.15.28:52014) 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e581a640 1 -- 172.21.15.28:0/3052833774 learned_addr learned my addr 172.21.15.28:0/3052833774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e5019640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e01a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.730+0000 7fa6e601b640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 0x7fa6e01a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 -- 172.21.15.28:0/3052833774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 msgr2=0x7fa6e01a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e01a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 -- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 msgr2=0x7fa6e01a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 0x7fa6e01a6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 -- 172.21.15.28:0/3052833774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e01a6770 con 0x7fa6e0104900 2024-09-21T05:53:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.731+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00bed0 tx=0x7fa6dc0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.732+0000 7fa6d6ffd640 1 -- 172.21.15.28:0/3052833774 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00db10 con 0x7fa6e0104900 2024-09-21T05:53:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.732+0000 7fa6d6ffd640 1 -- 172.21.15.28:0/3052833774 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc00dcb0 con 0x7fa6e0104900 2024-09-21T05:53:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.732+0000 7fa6d6ffd640 1 -- 172.21.15.28:0/3052833774 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc015620 con 0x7fa6e0104900 2024-09-21T05:53:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.732+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e01a6a60 con 0x7fa6e0104900 2024-09-21T05:53:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.732+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e01ad1f0 con 0x7fa6e0104900 2024-09-21T05:53:29.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.735+0000 7fa6ee9e7740 1 Processor -- start 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 -- start start 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 0x5630d65ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62202f0 0x5630d65eeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5630d65636f0 con 0x5630d62202f0 2024-09-21T05:53:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5630d6552b20 con 0x5630d62491b0 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.736+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5630d6556930 con 0x5630d62284d0 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e601b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 0x5630d65ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e601b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 0x5630d65ee9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36868/0 (socket says 172.21.15.28:36868) 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e601b640 1 -- 172.21.15.28:0/2942600885 learned_addr learned my addr 172.21.15.28:0/2942600885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 -- 172.21.15.28:0/2942600885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 msgr2=0x5630d65ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 0x5630d65ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 -- 172.21.15.28:0/2942600885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62202f0 msgr2=0x5630d65eeef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62202f0 0x5630d65eeef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.737+0000 7fa6e5019640 1 -- 172.21.15.28:0/2942600885 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630d6518530 con 0x5630d62284d0 2024-09-21T05:53:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.738+0000 7fa6e5019640 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ef430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa6e006c730 tx=0x7fa6d00056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39d2ca3f146d71d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.738+0000 7fa6d6ffd640 1 -- 172.21.15.28:0/3052833774 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa6dc016420 con 0x7fa6e0104900 2024-09-21T05:53:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.739+0000 7fa6cbfff640 1 -- 172.21.15.28:0/2942600885 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0037040 con 0x5630d62284d0 2024-09-21T05:53:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.740+0000 7fa6cbfff640 1 -- 172.21.15.28:0/2942600885 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d0030d50 con 0x5630d62284d0 2024-09-21T05:53:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.740+0000 7fa6d6ffd640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 0x7fa6cc0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.740+0000 7fa6d6ffd640 1 -- 172.21.15.28:0/3052833774 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa6dc0e74e0 con 0x7fa6e0104900 2024-09-21T05:53:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.740+0000 7fa6e601b640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 0x7fa6cc0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.741+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/2942600885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 msgr2=0x5630d65ef430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.741+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ef430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa6e006c730 tx=0x7fa6d00056d0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6e601b640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 0x7fa6cc0b8d60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e660 tx=0x7fa6d8010040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/2942600885 shutdown_connections 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ef430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62202f0 0x5630d65eeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/2942600885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62491b0 0x5630d65ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.745+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/2942600885 >> 172.21.15.28:0/2942600885 conn(0x5630d623d300 msgr2=0x5630d65ec8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.746+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/2942600885 shutdown_connections 2024-09-21T05:53:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.746+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/2942600885 wait complete. 2024-09-21T05:53:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.747+0000 7fa6ee9e7740 1 -- start start 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.747+0000 7fa6ee9e7740 1 Processor -- start 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62284d0 0x5630d65fa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d6606460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:52038/0 (socket says 172.21.15.28:52038) 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 -- 172.21.15.28:0/3248235800 learned_addr learned my addr 172.21.15.28:0/3248235800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5630d648aab0 con 0x5630d62491b0 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5630d64f8730 con 0x5630d62284d0 2024-09-21T05:53:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5630d65fd120 con 0x5630d62202f0 2024-09-21T05:53:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e5019640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62284d0 0x5630d65fa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6ee9e7740 10 client.0 _ll_register_callbacks cb 0x5630d654c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 -- 172.21.15.28:0/3248235800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62284d0 msgr2=0x5630d65fa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62284d0 0x5630d65fa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 -- 172.21.15.28:0/3248235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 msgr2=0x5630d6606460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d6606460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.748+0000 7fa6e581a640 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630d62b4790 con 0x5630d62202f0 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.749+0000 7fa6e581a640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00bea0 tx=0x7fa6dc0e50f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=290fcbea73ea9eb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.749+0000 7fa6c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.749+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0aca60 con 0x5630d62202f0 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.749+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d0030d50 con 0x5630d62202f0 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.749+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa6dc0abfc0 con 0x5630d62202f0 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.750+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa6dc0ace00 con 0x5630d62202f0 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.750+0000 7fa6ee9e7740 10 client.26632 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.750+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5630d632d620 con 0x5630d62202f0 2024-09-21T05:53:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.750+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc016e50 con 0x5630d62202f0 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.750+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fa6dc0b6250 con 0x5630d62202f0 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 10 client.26632 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 10 client.26632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 20 client.26632 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d6207d80 0x5630d660c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 4 client.26632 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa6d0030d50 con 0x5630d6207d80 2024-09-21T05:53:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.751+0000 7fa6e5019640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d6207d80 0x5630d660c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.753+0000 7fa6e5019640 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d6207d80 0x5630d660c690 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b711b6ef7bbe38b4 server_cookie=c22c0aeb2a54c097 in_seq=0 out_seq=0 2024-09-21T05:53:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.753+0000 7fa6c97fa640 10 client.26632 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.753+0000 7fa6c97fa640 1 -- 172.21.15.28:0/3248235800 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fa6d0030d50 con 0x5630d6207d80 2024-09-21T05:53:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.753+0000 7fa6c97fa640 10 client.26632 handle_command_reply: tid=0 2024-09-21T05:53:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.754+0000 7fa6ee9e7740 1 client.26632 shutdown 2024-09-21T05:53:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 msgr2=0x5630d65fa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00bea0 tx=0x7fa6dc0e50f0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 shutdown_connections 2024-09-21T05:53:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d6207d80 0x5630d660c690 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d6606460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62284d0 0x5630d65fa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/3248235800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62202f0 0x5630d65fa050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6c97fa640 0 client.26632 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.758+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 >> 172.21.15.28:0/3248235800 conn(0x5630d623d300 msgr2=0x5630d65ef9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.759+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 shutdown_connections 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.759+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/3248235800 wait complete. 2024-09-21T05:53:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.759+0000 7fa6ee9e7740 20 client.26632 trim_cache size 0 max 16384 2024-09-21T05:53:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.789+0000 7fa6ee9e7740 1 Processor -- start 2024-09-21T05:53:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.789+0000 7fa6ee9e7740 1 -- start start 2024-09-21T05:53:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d6606360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6e581a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36890/0 (socket says 172.21.15.28:36890) 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6e581a640 1 -- 172.21.15.28:0/4080083117 learned_addr learned my addr 172.21.15.28:0/4080083117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6e5019640 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d6606360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.790+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d66068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.791+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b5c90 con 0x5630d62491b0 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.791+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b7830 con 0x5630d62202f0 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.791+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b6360 con 0x5630d62284d0 2024-09-21T05:53:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.791+0000 7fa6e601b640 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d66068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 -- 172.21.15.28:0/4080083117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 msgr2=0x5630d6606360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d6606360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 -- 172.21.15.28:0/4080083117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 msgr2=0x5630d66068a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d66068a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 -- 172.21.15.28:0/4080083117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630d65f8530 con 0x5630d62202f0 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.792+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0aa560 tx=0x7fa6dc0b6780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8082816bef615d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.793+0000 7fa6cb7fe640 1 -- 172.21.15.28:0/4080083117 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0c7030 con 0x5630d62202f0 2024-09-21T05:53:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.793+0000 7fa6cb7fe640 1 -- 172.21.15.28:0/4080083117 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc016e50 con 0x5630d62202f0 2024-09-21T05:53:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.793+0000 7fa6cb7fe640 1 -- 172.21.15.28:0/4080083117 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0b6250 con 0x5630d62202f0 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 msgr2=0x5630d65f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0aa560 tx=0x7fa6dc0b6780 comp rx=0 tx=0).stop 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 shutdown_connections 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d66068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d6606360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4080083117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65f00f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 >> 172.21.15.28:0/4080083117 conn(0x5630d623d300 msgr2=0x5630d65ef9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 shutdown_connections 2024-09-21T05:53:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.794+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4080083117 wait complete. 2024-09-21T05:53:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.795+0000 7fa6ee9e7740 1 -- start start 2024-09-21T05:53:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.795+0000 7fa6ee9e7740 1 Processor -- start 2024-09-21T05:53:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65ff480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b7160 con 0x5630d62491b0 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b6a30 con 0x5630d62202f0 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.796+0000 7fa6ee9e7740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e01b3c70 con 0x5630d62284d0 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6ee9e7740 10 client.0 _ll_register_callbacks cb 0x7fa6dc009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40134/0 (socket says 172.21.15.28:40134) 2024-09-21T05:53:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 -- 172.21.15.28:0/4158568775 learned_addr learned my addr 172.21.15.28:0/4158568775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e5019640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 -- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 msgr2=0x5630d65ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 -- 172.21.15.28:0/4158568775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 msgr2=0x5630d65ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65ff480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 -- 172.21.15.28:0/4158568775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630d6675150 con 0x5630d62491b0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e5019640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6e601b640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800dfe0 tx=0x7fa6d80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48b7d99519b4561 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.797+0000 7fa6c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d802a020 con 0x5630d62491b0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fa6d8025670 con 0x5630d62491b0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc016e50 con 0x5630d62491b0 2024-09-21T05:53:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa6d8035050 con 0x5630d62491b0 2024-09-21T05:53:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6ee9e7740 10 client.17058 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5630d6676420 con 0x5630d62491b0 2024-09-21T05:53:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.798+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d8025d20 con 0x5630d62491b0 2024-09-21T05:53:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fa6d800b040 con 0x5630d62491b0 2024-09-21T05:53:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 10 client.17058 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 10 client.17058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 20 client.17058 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d61dd1c0 0x5630d6678c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 4 client.17058 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.799+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa6dc016e50 con 0x5630d61dd1c0 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.800+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d61dd1c0 0x5630d6678c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.800+0000 7fa6c97fa640 10 client.17058 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.800+0000 7fa6e581a640 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d61dd1c0 0x5630d6678c90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef01bb43f957b4f4 server_cookie=12f62ed0751e8edf in_seq=0 out_seq=0 2024-09-21T05:53:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6c97fa640 1 -- 172.21.15.28:0/4158568775 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa6dc016e50 con 0x5630d61dd1c0 2024-09-21T05:53:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6c97fa640 10 client.17058 handle_command_reply: tid=0 2024-09-21T05:53:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 client.17058 shutdown 2024-09-21T05:53:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 msgr2=0x5630d65fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800dfe0 tx=0x7fa6d80095c0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 shutdown_connections 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5630d61dd1c0 0x5630d6678c90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630d62491b0 0x5630d65fff00 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.824+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630d62284d0 0x5630d65ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.825+0000 7fa6ee9e7740 1 --2- 172.21.15.28:0/4158568775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630d62202f0 0x5630d65ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.825+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 >> 172.21.15.28:0/4158568775 conn(0x5630d623d300 msgr2=0x5630d65ec880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.825+0000 7fa6c97fa640 0 client.17058 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:29.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.826+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 shutdown_connections 2024-09-21T05:53:29.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.826+0000 7fa6ee9e7740 1 -- 172.21.15.28:0/4158568775 wait complete. 2024-09-21T05:53:29.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.826+0000 7fa6ee9e7740 20 client.17058 trim_cache size 0 max 16384 2024-09-21T05:53:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:53:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:53:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:53:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.828+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 msgr2=0x7fa6cc0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.828+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 0x7fa6cc0b8d60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e660 tx=0x7fa6d8010040 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.828+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 msgr2=0x7fa6e01a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.828+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00bed0 tx=0x7fa6dc0095b0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 shutdown_connections 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa6cc0b6850 0x7fa6cc0b8d60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e010dd60 0x7fa6e01a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6e0105250 0x7fa6e01a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 --2- 172.21.15.28:0/3052833774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6e0104900 0x7fa6e01a0840 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00bed0 tx=0x7fa6dc0095b0 comp rx=0 tx=0).stop 2024-09-21T05:53:29.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 >> 172.21.15.28:0/3052833774 conn(0x7fa6e007a740 msgr2=0x7fa6e0105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 shutdown_connections 2024-09-21T05:53:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:29.829+0000 7fa6e7a7e640 1 -- 172.21.15.28:0/3052833774 wait complete. 2024-09-21T05:53:29.911 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:30.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:30 smithi181 ceph-mon[34819]: pgmap v519: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 6.32k op/s 2024-09-21T05:53:30.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:30 smithi028 ceph-mon[31227]: pgmap v519: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 6.32k op/s 2024-09-21T05:53:30.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:30 smithi129 ceph-mon[34885]: pgmap v519: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 6.32k op/s 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/2641383213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 msgr2=0x7f7d78110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/2641383213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d78110630 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009ca0 tx=0x7f7d7002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/2641383213 shutdown_connections 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/2641383213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d78110630 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/2641383213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d781094b0 0x7f7d781098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/2641383213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78101f70 0x7f7d78102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/2641383213 >> 172.21.15.28:0/2641383213 conn(0x7f7d78077c60 msgr2=0x7f7d780780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/2641383213 shutdown_connections 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.494+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/2641383213 wait complete. 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.495+0000 7f7d7f0f1640 1 Processor -- start 2024-09-21T05:53:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.495+0000 7f7d7f0f1640 1 -- start start 2024-09-21T05:53:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 0x7f7d7819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d7819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113ac0 con 0x7f7d78109e00 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113940 con 0x7f7d78101f70 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7f0f1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113c40 con 0x7f7d781094b0 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d77fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 0x7f7d7819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36922/0 (socket says 172.21.15.28:36922) 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/929003820 learned_addr learned my addr 172.21.15.28:0/929003820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.496+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d7819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/929003820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 msgr2=0x7f7d7819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 0x7f7d7819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 msgr2=0x7f7d7819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d7819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/929003820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781a03e0 con 0x7f7d78101f70 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d77fff640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 0x7f7d7819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e9f0 tx=0x7f7d6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d75ffb640 1 -- 172.21.15.28:0/929003820 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00cd80 con 0x7f7d78101f70 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d781a06d0 con 0x7f7d78101f70 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.497+0000 7f7d75ffb640 1 -- 172.21.15.28:0/929003820 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d6c0108e0 con 0x7f7d78101f70 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.498+0000 7f7d75ffb640 1 -- 172.21.15.28:0/929003820 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c018980 con 0x7f7d78101f70 2024-09-21T05:53:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.498+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d781a6e30 con 0x7f7d78101f70 2024-09-21T05:53:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.501+0000 7f7d75ffb640 1 -- 172.21.15.28:0/929003820 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7d6c010430 con 0x7f7d78101f70 2024-09-21T05:53:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d75ffb640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 0x7f7d680b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 Processor -- start 2024-09-21T05:53:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 -- start start 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d77fff640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 0x7f7d680b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d75ffb640 1 -- 172.21.15.28:0/929003820 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7d6c0e79b0 con 0x7f7d78101f70 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab665be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab666120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649ab5dbea0 con 0x5649ab29d390 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649ab5cb2d0 con 0x5649ab2be070 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.503+0000 7f7d85665740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649ab5cf0e0 con 0x5649ab2951b0 2024-09-21T05:53:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7d68e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab666120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36928/0 (socket says 172.21.15.28:36928) 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3939074090 learned_addr learned my addr 172.21.15.28:0/3939074090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d77fff640 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab665be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d77fff640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 0x7f7d680b8c10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7d78107030 tx=0x7f7d6400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3939074090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 msgr2=0x5649ab665be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab665be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3939074090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 msgr2=0x5649ab666120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab666120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.504+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3939074090 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ab329650 con 0x5649ab2be070 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.505+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0a9ea0 tx=0x7f7d6c010720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a858ee2a2e32d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.505+0000 7f7d62ffd640 1 -- 172.21.15.28:0/3939074090 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0b2300 con 0x5649ab2be070 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.505+0000 7f7d62ffd640 1 -- 172.21.15.28:0/3939074090 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d6c0b7240 con 0x5649ab2be070 2024-09-21T05:53:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.505+0000 7f7d62ffd640 1 -- 172.21.15.28:0/3939074090 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0ac5a0 con 0x5649ab2be070 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.506+0000 7f7d85665740 1 -- 172.21.15.28:0/3939074090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 msgr2=0x5649ab6656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.506+0000 7f7d85665740 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0a9ea0 tx=0x7f7d6c010720 comp rx=0 tx=0).stop 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.507+0000 7f7d85665740 1 -- 172.21.15.28:0/3939074090 shutdown_connections 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.507+0000 7f7d85665740 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab666120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.507+0000 7f7d85665740 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab665be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.507+0000 7f7d85665740 1 --2- 172.21.15.28:0/3939074090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2be070 0x5649ab6656a0 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0a9ea0 tx=0x7f7d6c010720 comp rx=0 tx=0).stop 2024-09-21T05:53:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.507+0000 7f7d85665740 1 -- 172.21.15.28:0/3939074090 >> 172.21.15.28:0/3939074090 conn(0x5649ab2b2460 msgr2=0x5649ab663590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.508+0000 7f7d85665740 1 -- 172.21.15.28:0/3939074090 shutdown_connections 2024-09-21T05:53:30.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.509+0000 7f7d85665740 1 -- 172.21.15.28:0/3939074090 wait complete. 2024-09-21T05:53:30.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.510+0000 7f7d85665740 1 -- start start 2024-09-21T05:53:30.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.510+0000 7f7d85665740 1 Processor -- start 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d85665740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab670e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d85665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab671350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d85665740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab681520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d85665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649ab42c700 con 0x5649ab29d390 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab671350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab670e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab671350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40164/0 (socket says 172.21.15.28:40164) 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.511+0000 7f7d77fff640 1 -- 172.21.15.28:0/3035138863 learned_addr learned my addr 172.21.15.28:0/3035138863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab681520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649ab566160 con 0x5649ab27c9f0 2024-09-21T05:53:30.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5649ab678230 con 0x5649ab2951b0 2024-09-21T05:53:30.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d85665740 10 client.0 _ll_register_callbacks cb 0x5649ab57f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 msgr2=0x5649ab681520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab681520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3035138863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 msgr2=0x5649ab671350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab671350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.512+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ab593c90 con 0x5649ab2951b0 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.513+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab670e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0e56a0 tx=0x7f7d6c00cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=608d1a6b1d500f36 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.513+0000 7f7d57fff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.513+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0aed50 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.513+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7d6c018b20 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.513+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d6c0b7240 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7d6c0a9220 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 10 client.26647 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5649ab683fb0 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d6c0b7240 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f7d6c0b3490 con 0x5649ab2951b0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 10 client.26647 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 10 client.26647 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 20 client.26647 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab251da0 0x5649ab687140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 4 client.26647 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.514+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649ab5dddf0 con 0x5649ab251da0 2024-09-21T05:53:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.515+0000 7f7d77fff640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab251da0 0x5649ab687140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.516+0000 7f7d57fff640 10 client.26647 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:53:30.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.516+0000 7f7d77fff640 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab251da0 0x5649ab687140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52eb1bcd47318d8e server_cookie=57ac59a589b2fade in_seq=0 out_seq=0 2024-09-21T05:53:30.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d57fff640 1 -- 172.21.15.28:0/3035138863 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5649ab5dddf0 con 0x5649ab251da0 2024-09-21T05:53:30.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d57fff640 10 client.26647 handle_command_reply: tid=0 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 client.26647 shutdown 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 msgr2=0x5649ab670e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab670e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c0e56a0 tx=0x7f7d6c00cf20 comp rx=0 tx=0).stop 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 shutdown_connections 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab251da0 0x5649ab687140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab681520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab29d390 0x5649ab671350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 --2- 172.21.15.28:0/3035138863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab2951b0 0x5649ab670e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 >> 172.21.15.28:0/3035138863 conn(0x5649ab2b2460 msgr2=0x5649ab668880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.517+0000 7f7d57fff640 0 client.26647 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.518+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 shutdown_connections 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.518+0000 7f7d85665740 1 -- 172.21.15.28:0/3035138863 wait complete. 2024-09-21T05:53:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.518+0000 7f7d85665740 20 client.26647 trim_cache size 0 max 16384 2024-09-21T05:53:30.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.537+0000 7f7d85665740 1 Processor -- start 2024-09-21T05:53:30.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.537+0000 7f7d85665740 1 -- start start 2024-09-21T05:53:30.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d85665740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab668eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d85665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab2951b0 0x5649ab681490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab668eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d7ce8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab668eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36944/0 (socket says 172.21.15.28:36944) 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d7ce8d640 1 -- 172.21.15.28:0/417781263 learned_addr learned my addr 172.21.15.28:0/417781263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.538+0000 7f7d85665740 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab66e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781ad560 con 0x5649ab2951b0 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d77fff640 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab2951b0 0x5649ab681490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab66e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781af100 con 0x5649ab27c9f0 2024-09-21T05:53:30.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781adc30 con 0x5649ab29d390 2024-09-21T05:53:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 -- 172.21.15.28:0/417781263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 msgr2=0x5649ab668eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab668eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 -- 172.21.15.28:0/417781263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab2951b0 msgr2=0x5649ab681490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab2951b0 0x5649ab681490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.539+0000 7f7d7d68e640 1 -- 172.21.15.28:0/417781263 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ab667990 con 0x5649ab29d390 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.540+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab66e000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d70005e30 tx=0x7f7d70009440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34db508f019f2fda server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.540+0000 7f7d627fc640 1 -- 172.21.15.28:0/417781263 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70031450 con 0x5649ab29d390 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.540+0000 7f7d627fc640 1 -- 172.21.15.28:0/417781263 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d70036050 con 0x5649ab29d390 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.540+0000 7f7d627fc640 1 -- 172.21.15.28:0/417781263 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004310 con 0x5649ab29d390 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 msgr2=0x5649ab66e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab66e000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d70005e30 tx=0x7f7d70009440 comp rx=0 tx=0).stop 2024-09-21T05:53:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 shutdown_connections 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab66e000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab2951b0 0x5649ab681490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 --2- 172.21.15.28:0/417781263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab27c9f0 0x5649ab668eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 >> 172.21.15.28:0/417781263 conn(0x5649ab2b2460 msgr2=0x5649ab6687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.542+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 shutdown_connections 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.543+0000 7f7d85665740 1 -- 172.21.15.28:0/417781263 wait complete. 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.543+0000 7f7d85665740 1 -- start start 2024-09-21T05:53:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.544+0000 7f7d85665740 1 Processor -- start 2024-09-21T05:53:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.544+0000 7f7d85665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab27c9f0 0x5649ab6889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.544+0000 7f7d85665740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2951b0 0x5649ab688fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.544+0000 7f7d85665740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d85665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781aea30 con 0x5649ab27c9f0 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d85665740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781ae300 con 0x5649ab2951b0 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d85665740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781ab540 con 0x5649ab29d390 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d85665740 10 client.0 _ll_register_callbacks cb 0x7f7d70009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d7d68e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d7d68e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:52118/0 (socket says 172.21.15.28:52118) 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.545+0000 7f7d7d68e640 1 -- 172.21.15.28:0/2797163987 learned_addr learned my addr 172.21.15.28:0/2797163987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab27c9f0 0x5649ab6889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 -- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2951b0 msgr2=0x5649ab688fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2951b0 0x5649ab688fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 -- 172.21.15.28:0/2797163987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab27c9f0 msgr2=0x5649ab6889f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab27c9f0 0x5649ab6889f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 -- 172.21.15.28:0/2797163987 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ab6f0f30 con 0x5649ab29d390 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d7d68e640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009dc0 tx=0x7f7d7002a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce45eb9f67f5eb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.546+0000 7f7d57fff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.547+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7002f870 con 0x5649ab29d390 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.548+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7d7002fa10 con 0x5649ab29d390 2024-09-21T05:53:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.548+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d70036050 con 0x5649ab29d390 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.548+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7d7002fc10 con 0x5649ab29d390 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.548+0000 7f7d85665740 10 client.26653 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.548+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5649ab6f21b0 con 0x5649ab29d390 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d70036050 con 0x5649ab29d390 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f7d70031ce0 con 0x5649ab29d390 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 10 client.26653 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 10 client.26653 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:53:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 20 client.26653 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab6f2870 0x5649ab6f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 4 client.26653 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.549+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649ab66c9c0 con 0x5649ab6f2870 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.550+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab6f2870 0x5649ab6f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.551+0000 7f7d7ce8d640 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab6f2870 0x5649ab6f4c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6297d8be33cf8e4c server_cookie=6501b05d0fbe837 in_seq=0 out_seq=0 2024-09-21T05:53:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.551+0000 7f7d57fff640 10 client.26653 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:53:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.552+0000 7f7d57fff640 1 -- 172.21.15.28:0/2797163987 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649ab66c9c0 con 0x5649ab6f2870 2024-09-21T05:53:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.552+0000 7f7d57fff640 10 client.26653 handle_command_reply: tid=0 2024-09-21T05:53:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.552+0000 7f7d85665740 1 client.26653 shutdown 2024-09-21T05:53:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.553+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 msgr2=0x5649ab6f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.553+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009dc0 tx=0x7f7d7002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 shutdown_connections 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5649ab6f2870 0x5649ab6f4c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5649ab29d390 0x5649ab6f08c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649ab2951b0 0x5649ab688fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 --2- 172.21.15.28:0/2797163987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ab27c9f0 0x5649ab6889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 >> 172.21.15.28:0/2797163987 conn(0x5649ab2b2460 msgr2=0x5649ab668f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d57fff640 0 client.26653 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:53:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 shutdown_connections 2024-09-21T05:53:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 1 -- 172.21.15.28:0/2797163987 wait complete. 2024-09-21T05:53:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.554+0000 7f7d85665740 20 client.26653 trim_cache size 0 max 16384 2024-09-21T05:53:30.558 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:53:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.555+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 msgr2=0x7f7d680b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.555+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 0x7f7d680b8c10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7d78107030 tx=0x7f7d6400b410 comp rx=0 tx=0).stop 2024-09-21T05:53:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.556+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 msgr2=0x7f7d7819a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.556+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e9f0 tx=0x7f7d6c00eeb0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.556+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 shutdown_connections 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.556+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7d680b6700 0x7f7d680b8c10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.556+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d78109e00 0x7f7d7819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.557+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d781094b0 0x7f7d7819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.557+0000 7f7d7f0f1640 1 --2- 172.21.15.28:0/929003820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d78101f70 0x7f7d7819a470 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.557+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 >> 172.21.15.28:0/929003820 conn(0x7f7d78077c60 msgr2=0x7f7d78105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.557+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 shutdown_connections 2024-09-21T05:53:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:30.557+0000 7f7d7f0f1640 1 -- 172.21.15.28:0/929003820 wait complete. 2024-09-21T05:53:31.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:53:32.166 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:32.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:32 smithi181 ceph-mon[34819]: pgmap v520: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 45 KiB/s rd, 26 MiB/s wr, 6.12k op/s 2024-09-21T05:53:32.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:32 smithi028 ceph-mon[31227]: pgmap v520: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 45 KiB/s rd, 26 MiB/s wr, 6.12k op/s 2024-09-21T05:53:32.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:32 smithi129 ceph-mon[34885]: pgmap v520: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 45 KiB/s rd, 26 MiB/s wr, 6.12k op/s 2024-09-21T05:53:32.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.967+0000 7f4ebbfff640 1 -- 172.21.15.28:0/3886033749 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0030850 con 0x7f4ec4102870 2024-09-21T05:53:32.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 -- 172.21.15.28:0/3886033749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4102870 msgr2=0x7f4ec4102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:32.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 --2- 172.21.15.28:0/3886033749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4102870 0x7f4ec4102c80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009860 tx=0x7f4ec002a340 comp rx=0 tx=0).stop 2024-09-21T05:53:32.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 -- 172.21.15.28:0/3886033749 shutdown_connections 2024-09-21T05:53:32.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 --2- 172.21.15.28:0/3886033749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4108b00 0x7f4ec410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 --2- 172.21.15.28:0/3886033749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 0x7f4ec41085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 --2- 172.21.15.28:0/3886033749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4102870 0x7f4ec4102c80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.968+0000 7f4eca22c640 1 -- 172.21.15.28:0/3886033749 >> 172.21.15.28:0/3886033749 conn(0x7f4ec40fe2c0 msgr2=0x7f4ec4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:32.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.972+0000 7f4eca22c640 1 -- 172.21.15.28:0/3886033749 shutdown_connections 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.972+0000 7f4eca22c640 1 -- 172.21.15.28:0/3886033749 wait complete. 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 Processor -- start 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 -- start start 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 0x7f4ec41138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 0x7f4ec4118bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41116e0 con 0x7f4ec41081b0 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4111560 con 0x7f4ec4102870 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.973+0000 7f4eca22c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4111860 con 0x7f4ec4108b00 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec9a2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 0x7f4ec4118bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36980/0 (socket says 172.21.15.28:36980) 2024-09-21T05:53:32.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 -- 172.21.15.28:0/463457799 learned_addr learned my addr 172.21.15.28:0/463457799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:32.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 0x7f4ec41138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 -- 172.21.15.28:0/463457799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 msgr2=0x7f4ec4118bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 0x7f4ec4118bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 -- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 msgr2=0x7f4ec41138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 0x7f4ec41138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.974+0000 7f4ec922a640 1 -- 172.21.15.28:0/463457799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec41192e0 con 0x7f4ec4102870 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.975+0000 7f4ec922a640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0035040 tx=0x7f4ec0032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.975+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 0x7f4ec4118bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.976+0000 7f4eba7fc640 1 -- 172.21.15.28:0/463457799 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0037040 con 0x7f4ec4102870 2024-09-21T05:53:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.976+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec4119510 con 0x7f4ec4102870 2024-09-21T05:53:32.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.976+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec4119990 con 0x7f4ec4102870 2024-09-21T05:53:32.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.976+0000 7f4eba7fc640 1 -- 172.21.15.28:0/463457799 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec0041070 con 0x7f4ec4102870 2024-09-21T05:53:32.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.977+0000 7f4eba7fc640 1 -- 172.21.15.28:0/463457799 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00035f0 con 0x7f4ec4102870 2024-09-21T05:53:32.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.980+0000 7f4ed1a02740 1 Processor -- start 2024-09-21T05:53:32.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 -- start start 2024-09-21T05:53:32.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x561412120860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x561412120da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 0x5614121212e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561412096ea0 con 0x561411d501b0 2024-09-21T05:53:32.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.981+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5614120862d0 con 0x561411d58390 2024-09-21T05:53:32.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.983+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56141208a0e0 con 0x561411d79070 2024-09-21T05:53:32.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.983+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x561412120da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.984+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x561412120da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40212/0 (socket says 172.21.15.28:40212) 2024-09-21T05:53:32.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.984+0000 7f4ec922a640 1 -- 172.21.15.28:0/1371890766 learned_addr learned my addr 172.21.15.28:0/1371890766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:32.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.982+0000 7f4ec8a29640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 0x5614121212e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.982+0000 7f4ec9a2b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x561412120860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.983+0000 7f4eba7fc640 1 -- 172.21.15.28:0/463457799 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f4ec0004050 con 0x7f4ec4102870 2024-09-21T05:53:32.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 -- 172.21.15.28:0/1371890766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 msgr2=0x561412120860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:32.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x561412120860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 -- 172.21.15.28:0/1371890766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 msgr2=0x561412120da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:32.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x561412120da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 -- 172.21.15.28:0/1371890766 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561411de4650 con 0x561411d58390 2024-09-21T05:53:32.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.985+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 0x5614121212e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ec406a800 tx=0x7f4eb40113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0c58482cbf5f5a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:32.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.986+0000 7f4eaf7fe640 1 -- 172.21.15.28:0/1371890766 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4011de0 con 0x561411d58390 2024-09-21T05:53:32.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.986+0000 7f4eaf7fe640 1 -- 172.21.15.28:0/1371890766 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40054d0 con 0x561411d58390 2024-09-21T05:53:32.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.986+0000 7f4eaf7fe640 1 -- 172.21.15.28:0/1371890766 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40126b0 con 0x561411d58390 2024-09-21T05:53:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.987+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1371890766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 msgr2=0x5614121212e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.987+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 0x5614121212e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ec406a800 tx=0x7f4eb40113e0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.988+0000 7f4eba7fc640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 0x7f4eb00b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.988+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 0x7f4eb00b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1371890766 shutdown_connections 2024-09-21T05:53:32.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d58390 0x5614121212e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x561412120da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1371890766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x561412120860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1371890766 >> 172.21.15.28:0/1371890766 conn(0x561411d6d460 msgr2=0x56141211e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 0x7f4eb00b8ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc005fd0 tx=0x7f4ebc009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4eba7fc640 1 -- 172.21.15.28:0/463457799 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4ec0107130 con 0x7f4ec4102870 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1371890766 shutdown_connections 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.989+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1371890766 wait complete. 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.990+0000 7f4ed1a02740 1 -- start start 2024-09-21T05:53:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.990+0000 7f4ed1a02740 1 Processor -- start 2024-09-21T05:53:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 0x56141212bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x56141212c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x5614121493d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561411ee7700 con 0x561411d58390 2024-09-21T05:53:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561412021160 con 0x561411d79070 2024-09-21T05:53:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x561412140050 con 0x561411d501b0 2024-09-21T05:53:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ed1a02740 10 client.0 _ll_register_callbacks cb 0x56141203a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ec8a29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x56141212c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ec8a29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x56141212c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40228/0 (socket says 172.21.15.28:40228) 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ec8a29640 1 -- 172.21.15.28:0/507839656 learned_addr learned my addr 172.21.15.28:0/507839656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ec922a640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 0x56141212bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.991+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x5614121493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 -- 172.21.15.28:0/507839656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 msgr2=0x5614121493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x5614121493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 -- 172.21.15.28:0/507839656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 msgr2=0x56141212c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x56141212c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 -- 172.21.15.28:0/507839656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56141204ec90 con 0x561411d501b0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ec922a640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 0x56141212bf30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x56141211f4e0 tx=0x7f4ec010a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70bb5bd3f627c329 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4eacff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00ccdf0 con 0x561411d501b0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb40054d0 con 0x561411d501b0 2024-09-21T05:53:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4ec00cc480 con 0x561411d501b0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.992+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4ec00cc790 con 0x561411d501b0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 10 client.26665 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561412130540 con 0x561411d501b0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40054d0 con 0x561411d501b0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f4ec00b8190 con 0x561411d501b0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 10 client.26665 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 10 client.26665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 20 client.26665 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d379f0 0x561412133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.993+0000 7f4ed1a02740 4 client.26665 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.994+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561412098df0 con 0x561411d379f0 2024-09-21T05:53:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.994+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d379f0 0x561412133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.994+0000 7f4eacff9640 10 client.26665 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.995+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d379f0 0x561412133c70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95b27aee9a81f989 server_cookie=5fdcf3d4c6a5e380 in_seq=0 out_seq=0 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4eacff9640 1 -- 172.21.15.28:0/507839656 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561412098df0 con 0x561411d379f0 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4eacff9640 10 client.26665 handle_command_reply: tid=0 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 client.26665 shutdown 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 msgr2=0x56141212bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 0x56141212bf30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x56141211f4e0 tx=0x7f4ec010a790 comp rx=0 tx=0).stop 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 shutdown_connections 2024-09-21T05:53:32.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d379f0 0x561412133c70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x5614121493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x56141212c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/507839656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d501b0 0x56141212bf30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 >> 172.21.15.28:0/507839656 conn(0x561411d6d460 msgr2=0x5614121239f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4eacff9640 0 client.26665 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 shutdown_connections 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.996+0000 7f4ed1a02740 1 -- 172.21.15.28:0/507839656 wait complete. 2024-09-21T05:53:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:32.997+0000 7f4ed1a02740 20 client.26665 trim_cache size 0 max 16384 2024-09-21T05:53:33.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.017+0000 7f4ed1a02740 1 Processor -- start 2024-09-21T05:53:33.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.017+0000 7f4ed1a02740 1 -- start start 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d58390 0x5614121492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x561412123960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea800e860 con 0x561411d501b0 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea8010400 con 0x561411d79070 2024-09-21T05:53:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea800ef30 con 0x561411d58390 2024-09-21T05:53:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40240/0 (socket says 172.21.15.28:40240) 2024-09-21T05:53:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ec922a640 1 -- 172.21.15.28:0/2924148831 learned_addr learned my addr 172.21.15.28:0/2924148831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d58390 0x5614121492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.018+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x561412123960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/2924148831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d58390 msgr2=0x5614121492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d58390 0x5614121492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/2924148831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 msgr2=0x56141212c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/2924148831 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561412122b50 con 0x561411d79070 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec922a640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x561412123960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00f8d0 tx=0x7f4ebc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24729fc0b0e257d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4eaeffd640 1 -- 172.21.15.28:0/2924148831 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc004d20 con 0x561411d79070 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4eaeffd640 1 -- 172.21.15.28:0/2924148831 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ebc01f980 con 0x561411d79070 2024-09-21T05:53:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.019+0000 7f4eaeffd640 1 -- 172.21.15.28:0/2924148831 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc005460 con 0x561411d79070 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.020+0000 7f4ed1a02740 1 -- 172.21.15.28:0/2924148831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 msgr2=0x561412123960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.020+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x561412123960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00f8d0 tx=0x7f4ebc016000 comp rx=0 tx=0).stop 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 -- 172.21.15.28:0/2924148831 shutdown_connections 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d79070 0x561412123960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d58390 0x5614121492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/2924148831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d501b0 0x56141212c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 -- 172.21.15.28:0/2924148831 >> 172.21.15.28:0/2924148831 conn(0x561411d6d460 msgr2=0x56141212bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:33.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 -- 172.21.15.28:0/2924148831 shutdown_connections 2024-09-21T05:53:33.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.021+0000 7f4ed1a02740 1 -- 172.21.15.28:0/2924148831 wait complete. 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.022+0000 7f4ed1a02740 1 -- start start 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.022+0000 7f4ed1a02740 1 Processor -- start 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.022+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.022+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x561412141d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.022+0000 7f4ed1a02740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x5614121c9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea800fd30 con 0x561411d58390 2024-09-21T05:53:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea800f600 con 0x561411d501b0 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ed1a02740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea800c840 con 0x561411d79070 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ed1a02740 10 client.0 _ll_register_callbacks cb 0x7f4ec0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ec922a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:36996/0 (socket says 172.21.15.28:36996) 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ec922a640 1 -- 172.21.15.28:0/1179273882 learned_addr learned my addr 172.21.15.28:0/1179273882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.023+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x5614121c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x561412141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/1179273882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 msgr2=0x561412141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/1179273882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 msgr2=0x561412141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x561412141d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 -- 172.21.15.28:0/1179273882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614121c9c70 con 0x561411d79070 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec922a640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec8a29640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x561412141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4ec9a2b640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x5614121c9730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x56141211a110 tx=0x7f4ebc01f8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d8057cc5408bf2e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.024+0000 7f4eacff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0050c0 con 0x561411d79070 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f4ebc005260 con 0x561411d79070 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ebc01f980 con 0x561411d79070 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4ebc005460 con 0x561411d79070 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4ed1a02740 10 client.26671 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.025+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5614121cadf0 con 0x561411d79070 2024-09-21T05:53:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.026+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ebc01f980 con 0x561411d79070 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.027+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f4ebc031070 con 0x561411d79070 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 10 client.26671 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 10 client.26671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 20 client.26671 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d0cda0 0x5614121cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 4 client.26671 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:53:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614120948a0 con 0x561411d0cda0 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.028+0000 7f4ec922a640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d0cda0 0x5614121cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ec922a640 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d0cda0 0x5614121cd470 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf0831c069d87fe server_cookie=ad3f6b738853e8c9 in_seq=0 out_seq=0 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4eacff9640 10 client.26671 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4eacff9640 1 -- 172.21.15.28:0/1179273882 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614120948a0 con 0x561411d0cda0 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4eacff9640 10 client.26671 handle_command_reply: tid=0 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 client.26671 shutdown 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 msgr2=0x5614121c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x5614121c9730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x56141211a110 tx=0x7f4ebc01f8c0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 shutdown_connections 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x561411d0cda0 0x5614121cd470 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561411d79070 0x5614121c9730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4eacff9640 0 client.26671 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561411d58390 0x561412141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 --2- 172.21.15.28:0/1179273882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561411d501b0 0x561412141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.029+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 >> 172.21.15.28:0/1179273882 conn(0x561411d6d460 msgr2=0x561412123960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.030+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 shutdown_connections 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.030+0000 7f4ed1a02740 1 -- 172.21.15.28:0/1179273882 wait complete. 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.030+0000 7f4ed1a02740 20 client.26671 trim_cache size 0 max 16384 2024-09-21T05:53:33.033 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:53:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.033+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 msgr2=0x7f4eb00b8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.033+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 0x7f4eb00b8ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc005fd0 tx=0x7f4ebc009500 comp rx=0 tx=0).stop 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.033+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 msgr2=0x7f4ec4113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0035040 tx=0x7f4ec0032d50 comp rx=0 tx=0).stop 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 shutdown_connections 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4108b00 0x7f4ec4118bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f4eb00b65b0 0x7f4eb00b8ac0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ec41081b0 0x7f4ec41138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 --2- 172.21.15.28:0/463457799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ec4102870 0x7f4ec4113370 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 >> 172.21.15.28:0/463457799 conn(0x7f4ec40fe2c0 msgr2=0x7f4ec41006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 shutdown_connections 2024-09-21T05:53:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:33.034+0000 7f4eca22c640 1 -- 172.21.15.28:0/463457799 wait complete. 2024-09-21T05:53:34.293 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:34 smithi181 ceph-mon[34819]: pgmap v521: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 6.13k op/s 2024-09-21T05:53:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:34 smithi028 ceph-mon[31227]: pgmap v521: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 6.13k op/s 2024-09-21T05:53:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:34 smithi129 ceph-mon[34885]: pgmap v521: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 6.13k op/s 2024-09-21T05:53:35.158 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:53:35.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:53:35.538 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 -- 172.21.15.28:0/3961077399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd010a9e0 msgr2=0x7f7dd010adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 --2- 172.21.15.28:0/3961077399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd010a9e0 0x7f7dd010adf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009860 tx=0x7f7dcc02a380 comp rx=0 tx=0).stop 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 -- 172.21.15.28:0/3961077399 shutdown_connections 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 --2- 172.21.15.28:0/3961077399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd011cc30 0x7f7dd011f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 --2- 172.21.15.28:0/3961077399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd010a9e0 0x7f7dd010adf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 --2- 172.21.15.28:0/3961077399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd0111f40 0x7f7dd010a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.177+0000 7f7dd797f640 1 -- 172.21.15.28:0/3961077399 >> 172.21.15.28:0/3961077399 conn(0x7f7dd0070ec0 msgr2=0x7f7dd00712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.183+0000 7f7dd797f640 1 -- 172.21.15.28:0/3961077399 shutdown_connections 2024-09-21T05:53:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.183+0000 7f7dd797f640 1 -- 172.21.15.28:0/3961077399 wait complete. 2024-09-21T05:53:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.184+0000 7f7dd797f640 1 Processor -- start 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 -- start start 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd010a9e0 0x7f7dd011a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 0x7f7dd0113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 0x7f7dd0113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd0122960 con 0x7f7dd011cc30 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01227e0 con 0x7f7dd010a9e0 2024-09-21T05:53:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.190+0000 7f7dd797f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd0122ae0 con 0x7f7dd0111f40 2024-09-21T05:53:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.192+0000 7f7dd571b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd010a9e0 0x7f7dd011a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.192+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 0x7f7dd0113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.192+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 0x7f7dd0113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50732/0 (socket says 172.21.15.28:50732) 2024-09-21T05:53:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.192+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/4293250694 learned_addr learned my addr 172.21.15.28:0/4293250694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.192+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 0x7f7dd0113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.194+0000 7f7dd4f1a640 1 -- 172.21.15.28:0/4293250694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd010a9e0 msgr2=0x7f7dd011a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.194+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd010a9e0 0x7f7dd011a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.194+0000 7f7dd4f1a640 1 -- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 msgr2=0x7f7dd0113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.194+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 0x7f7dd0113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.194+0000 7f7dd4f1a640 1 -- 172.21.15.28:0/4293250694 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd01191b0 con 0x7f7dd0111f40 2024-09-21T05:53:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.201+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 0x7f7dd0113810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009780 tx=0x7f7dcc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.204+0000 7f7dc67fc640 1 -- 172.21.15.28:0/4293250694 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc037050 con 0x7f7dd0111f40 2024-09-21T05:53:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.204+0000 7f7dc67fc640 1 -- 172.21.15.28:0/4293250694 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dcc038040 con 0x7f7dd0111f40 2024-09-21T05:53:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.204+0000 7f7dc67fc640 1 -- 172.21.15.28:0/4293250694 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc030a70 con 0x7f7dd0111f40 2024-09-21T05:53:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.204+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd0119440 con 0x7f7dd0111f40 2024-09-21T05:53:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.205+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd01198a0 con 0x7f7dd0111f40 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.212+0000 7f7dddef3740 1 Processor -- start 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.217+0000 7f7dc67fc640 1 -- 172.21.15.28:0/4293250694 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7dcc045070 con 0x7f7dd0111f40 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 -- start start 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90be4670 0x562f90faceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f90f22e60 con 0x562f90c05350 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562f90f12290 con 0x562f90bdc490 2024-09-21T05:53:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.223+0000 7f7dddef3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562f90f160a0 con 0x562f90be4670 2024-09-21T05:53:36.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.225+0000 7f7dc67fc640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 0x7f7dbc0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.226+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.226+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50744/0 (socket says 172.21.15.28:50744) 2024-09-21T05:53:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.226+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2957609574 learned_addr learned my addr 172.21.15.28:0/2957609574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.226+0000 7f7dc67fc640 1 -- 172.21.15.28:0/4293250694 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7dcc106e50 con 0x7f7dd0111f40 2024-09-21T05:53:36.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.228+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90be4670 0x562f90faceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 0x7f7dbc0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2957609574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90be4670 msgr2=0x562f90faceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90be4670 0x562f90faceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2957609574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 msgr2=0x562f90fac970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.229+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2957609574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f90c70930 con 0x562f90c05350 2024-09-21T05:53:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.230+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009c40 tx=0x7f7dc800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a5ccd8c5c0fa01 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.230+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 0x7f7dbc0b8d10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800dcb0 tx=0x7f7dc801c000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.231+0000 7f7dbb7fe640 1 -- 172.21.15.28:0/2957609574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc801be00 con 0x562f90c05350 2024-09-21T05:53:36.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.231+0000 7f7dbb7fe640 1 -- 172.21.15.28:0/2957609574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc801c8c0 con 0x562f90c05350 2024-09-21T05:53:36.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.231+0000 7f7dbb7fe640 1 -- 172.21.15.28:0/2957609574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8024630 con 0x562f90c05350 2024-09-21T05:53:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 -- 172.21.15.28:0/2957609574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 msgr2=0x562f90fac430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009c40 tx=0x7f7dc800d7b0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 -- 172.21.15.28:0/2957609574 shutdown_connections 2024-09-21T05:53:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90be4670 0x562f90faceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2957609574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90fac430 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.234+0000 7f7dddef3740 1 -- 172.21.15.28:0/2957609574 >> 172.21.15.28:0/2957609574 conn(0x562f90bf9740 msgr2=0x562f90faa320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.239+0000 7f7dddef3740 1 -- 172.21.15.28:0/2957609574 shutdown_connections 2024-09-21T05:53:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.241+0000 7f7dddef3740 1 -- 172.21.15.28:0/2957609574 wait complete. 2024-09-21T05:53:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.241+0000 7f7dddef3740 1 -- start start 2024-09-21T05:53:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.241+0000 7f7dddef3740 1 Processor -- start 2024-09-21T05:53:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fb8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fb86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f90d73890 con 0x562f90be4670 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562f90eacd40 con 0x562f90bdc490 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562f90fbabb0 con 0x562f90c05350 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.242+0000 7f7dddef3740 10 client.0 _ll_register_callbacks cb 0x562f90f10c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.243+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.243+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44158/0 (socket says 172.21.15.28:44158) 2024-09-21T05:53:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.243+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/4052643632 learned_addr learned my addr 172.21.15.28:0/4052643632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.244+0000 7f7dd571b640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fb8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:36 smithi028 ceph-mon[31227]: pgmap v522: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 44 KiB/s rd, 19 MiB/s wr, 4.52k op/s 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fb86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/4052643632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 msgr2=0x562f90fb8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fb8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/4052643632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 msgr2=0x562f90fb86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fb86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/4052643632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f90fc4510 con 0x562f90c05350 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fb86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.247+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009df0 tx=0x7f7dc801bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c6ba8fcaed66bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.251+0000 7f7db8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.251+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc802a040 con 0x562f90c05350 2024-09-21T05:53:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.251+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc801c8c0 con 0x562f90c05350 2024-09-21T05:53:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.251+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7dc801c610 con 0x562f90c05350 2024-09-21T05:53:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.256+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7dc801ca70 con 0x562f90c05350 2024-09-21T05:53:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.256+0000 7f7dddef3740 10 client.26683 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.256+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562f90fc6e40 con 0x562f90c05350 2024-09-21T05:53:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc801c8c0 con 0x562f90c05350 2024-09-21T05:53:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f7dc8036030 con 0x562f90c05350 2024-09-21T05:53:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 10 client.26683 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 10 client.26683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 20 client.26683 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90bc3cd0 0x562f90fca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 4 client.26683 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.260+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f90f24db0 con 0x562f90bc3cd0 2024-09-21T05:53:36.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.264+0000 7f7dd571b640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90bc3cd0 0x562f90fca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.269+0000 7f7dd571b640 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90bc3cd0 0x562f90fca570 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3a13e8ecfdbebf server_cookie=9c4b0d7f959cd2dd in_seq=0 out_seq=0 2024-09-21T05:53:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.269+0000 7f7db8ff9640 10 client.26683 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.274+0000 7f7db8ff9640 1 -- 172.21.15.28:0/4052643632 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562f90f24db0 con 0x562f90bc3cd0 2024-09-21T05:53:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.274+0000 7f7db8ff9640 10 client.26683 handle_command_reply: tid=0 2024-09-21T05:53:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.275+0000 7f7dddef3740 1 client.26683 shutdown 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 msgr2=0x562f90fc3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009df0 tx=0x7f7dc801bce0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 shutdown_connections 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90bc3cd0 0x562f90fca570 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fc3ea0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fb86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 --2- 172.21.15.28:0/4052643632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fb8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 >> 172.21.15.28:0/4052643632 conn(0x562f90bf9740 msgr2=0x562f90faf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.278+0000 7f7db8ff9640 0 client.26683 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.280+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 shutdown_connections 2024-09-21T05:53:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.280+0000 7f7dddef3740 1 -- 172.21.15.28:0/4052643632 wait complete. 2024-09-21T05:53:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.281+0000 7f7dddef3740 20 client.26683 trim_cache size 0 max 16384 2024-09-21T05:53:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.318+0000 7f7dddef3740 1 Processor -- start 2024-09-21T05:53:36.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 -- start start 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90bdc490 0x562f90fc3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90be4670 0x562f90fa9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c6eb0 con 0x562f90c05350 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c8a50 con 0x562f90be4670 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.322+0000 7f7dddef3740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c7580 con 0x562f90bdc490 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.323+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.323+0000 7f7dd5f1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50768/0 (socket says 172.21.15.28:50768) 2024-09-21T05:53:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.323+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2525913521 learned_addr learned my addr 172.21.15.28:0/2525913521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd571b640 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90bdc490 0x562f90fc3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90be4670 0x562f90fa9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2525913521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90bdc490 msgr2=0x562f90fc3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90bdc490 0x562f90fc3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2525913521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90be4670 msgr2=0x562f90fa9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90be4670 0x562f90fa9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.332+0000 7f7dd5f1c640 1 -- 172.21.15.28:0/2525913521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f90fb6e70 con 0x562f90c05350 2024-09-21T05:53:36.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.337+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009c40 tx=0x7f7dc8005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e7178175d7ad2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.341+0000 7f7dbaffd640 1 -- 172.21.15.28:0/2525913521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc803f070 con 0x562f90c05350 2024-09-21T05:53:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.341+0000 7f7dbaffd640 1 -- 172.21.15.28:0/2525913521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc8005000 con 0x562f90c05350 2024-09-21T05:53:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.341+0000 7f7dbaffd640 1 -- 172.21.15.28:0/2525913521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8003bd0 con 0x562f90c05350 2024-09-21T05:53:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.343+0000 7f7dddef3740 1 -- 172.21.15.28:0/2525913521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 msgr2=0x562f90faa1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.343+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009c40 tx=0x7f7dc8005c70 comp rx=0 tx=0).stop 2024-09-21T05:53:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.345+0000 7f7dddef3740 1 -- 172.21.15.28:0/2525913521 shutdown_connections 2024-09-21T05:53:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.345+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90c05350 0x562f90faa1a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.345+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90be4670 0x562f90fa9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.345+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2525913521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90bdc490 0x562f90fc3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.345+0000 7f7dddef3740 1 -- 172.21.15.28:0/2525913521 >> 172.21.15.28:0/2525913521 conn(0x562f90bf9740 msgr2=0x562f90fb30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.347+0000 7f7dddef3740 1 -- 172.21.15.28:0/2525913521 shutdown_connections 2024-09-21T05:53:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.347+0000 7f7dddef3740 1 -- 172.21.15.28:0/2525913521 wait complete. 2024-09-21T05:53:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.348+0000 7f7dddef3740 1 -- start start 2024-09-21T05:53:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.348+0000 7f7dddef3740 1 Processor -- start 2024-09-21T05:53:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.350+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.350+0000 7f7dddef3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fc0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.350+0000 7f7dd571b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.350+0000 7f7dd571b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47200/0 (socket says 172.21.15.28:47200) 2024-09-21T05:53:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.350+0000 7f7dd571b640 1 -- 172.21.15.28:0/2874114579 learned_addr learned my addr 172.21.15.28:0/2874114579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.351+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fbadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.351+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c8380 con 0x562f90be4670 2024-09-21T05:53:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.351+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fbadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.351+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c7c50 con 0x562f90bdc490 2024-09-21T05:53:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.351+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd01c4e90 con 0x562f90c05350 2024-09-21T05:53:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.352+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fc0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.352+0000 7f7dddef3740 10 client.0 _ll_register_callbacks cb 0x7f7dc800e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.354+0000 7f7dd571b640 1 -- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 msgr2=0x562f90fbadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.354+0000 7f7dd571b640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fbadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.354+0000 7f7dd571b640 1 -- 172.21.15.28:0/2874114579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 msgr2=0x562f90fc0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.354+0000 7f7dd571b640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fc0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.354+0000 7f7dd571b640 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f90fbb300 con 0x562f90bdc490 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.356+0000 7f7dd571b640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x562f90fcac90 tx=0x7f7dc0016790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b338eabcee9556a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.356+0000 7f7db8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.356+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0020070 con 0x562f90bdc490 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.356+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f7dc0017630 con 0x562f90bdc490 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.356+0000 7f7dd5f1c640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fbadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.357+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fc0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.357+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc8005000 con 0x562f90bdc490 2024-09-21T05:53:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.357+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7dc002a050 con 0x562f90bdc490 2024-09-21T05:53:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.358+0000 7f7dddef3740 10 client.26681 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.358+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562f91035640 con 0x562f90bdc490 2024-09-21T05:53:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc8005000 con 0x562f90bdc490 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f7dc00257f0 con 0x562f90bdc490 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7dddef3740 10 client.26681 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7dddef3740 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7dddef3740 20 client.26681 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.360+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90b99080 0x562f91037e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.361+0000 7f7dddef3740 4 client.26681 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.361+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562f90f20860 con 0x562f90b99080 2024-09-21T05:53:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.362+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90b99080 0x562f91037e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:36.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:36 smithi129 ceph-mon[34885]: pgmap v522: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 44 KiB/s rd, 19 MiB/s wr, 4.52k op/s 2024-09-21T05:53:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.365+0000 7f7db8ff9640 10 client.26681 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.365+0000 7f7dd4f1a640 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90b99080 0x562f91037e30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8da29fbd28f85c5 server_cookie=1c35f141c20ce1ab in_seq=0 out_seq=0 2024-09-21T05:53:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.370+0000 7f7db8ff9640 1 -- 172.21.15.28:0/2874114579 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562f90f20860 con 0x562f90b99080 2024-09-21T05:53:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.370+0000 7f7db8ff9640 10 client.26681 handle_command_reply: tid=0 2024-09-21T05:53:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.370+0000 7f7dddef3740 1 client.26681 shutdown 2024-09-21T05:53:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 msgr2=0x562f90fbfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x562f90fcac90 tx=0x7f7dc0016790 comp rx=0 tx=0).stop 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 shutdown_connections 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x562f90b99080 0x562f91037e30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562f90c05350 0x562f90fbadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7db8ff9640 0 client.26681 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f90be4670 0x562f90fc0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 --2- 172.21.15.28:0/2874114579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562f90bdc490 0x562f90fbfe10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.373+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 >> 172.21.15.28:0/2874114579 conn(0x562f90bf9740 msgr2=0x562f90fa9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.377+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 shutdown_connections 2024-09-21T05:53:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.378+0000 7f7dddef3740 1 -- 172.21.15.28:0/2874114579 wait complete. 2024-09-21T05:53:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.378+0000 7f7dddef3740 20 client.26681 trim_cache size 0 max 16384 2024-09-21T05:53:36.385 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:53:36.385 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:53:36.385 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b3760c46-b2b6-415c-99f0-96d4557921a0", 2024-09-21T05:53:36.385 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:53:36.385 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:53:36.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 msgr2=0x7f7dbc0b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 0x7f7dbc0b8d10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800dcb0 tx=0x7f7dc801c000 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 msgr2=0x7f7dd0113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 0x7f7dd0113810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009780 tx=0x7f7dcc004270 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 shutdown_connections 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7dbc0b6800 0x7f7dbc0b8d10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd011cc30 0x7f7dd0113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7dd0111f40 0x7f7dd0113810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 --2- 172.21.15.28:0/4293250694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dd010a9e0 0x7f7dd011a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.384+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 >> 172.21.15.28:0/4293250694 conn(0x7f7dd0070ec0 msgr2=0x7f7dd011e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.385+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 shutdown_connections 2024-09-21T05:53:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:36.385+0000 7f7dd797f640 1 -- 172.21.15.28:0/4293250694 wait complete. 2024-09-21T05:53:36.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:36 smithi181 ceph-mon[34819]: pgmap v522: 97 pgs: 97 active+clean; 18 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 44 KiB/s rd, 19 MiB/s wr, 4.52k op/s 2024-09-21T05:53:37.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:53:37.663 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:53:38.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: pgmap v523: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:53:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:38 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:53:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:53:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:53:38.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: pgmap v523: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-09-21T05:53:38.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:53:38.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:53:38.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:53:38.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:38 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: pgmap v523: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 43 KiB/s rd, 17 MiB/s wr, 4.12k op/s 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:53:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:38 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.860+0000 7fb97d85e640 1 -- 172.21.15.28:0/4087198936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810a650 msgr2=0x7fb97810aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.860+0000 7fb97d85e640 1 --2- 172.21.15.28:0/4087198936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810a650 0x7fb97810aa60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00af00 tx=0x7fb96c0329f0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.862+0000 7fb97d85e640 1 -- 172.21.15.28:0/4087198936 shutdown_connections 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.862+0000 7fb97d85e640 1 --2- 172.21.15.28:0/4087198936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810d620 0x7fb978117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.862+0000 7fb97d85e640 1 --2- 172.21.15.28:0/4087198936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb97810d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.862+0000 7fb97d85e640 1 --2- 172.21.15.28:0/4087198936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810a650 0x7fb97810aa60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.862+0000 7fb97d85e640 1 -- 172.21.15.28:0/4087198936 >> 172.21.15.28:0/4087198936 conn(0x7fb978070ec0 msgr2=0x7fb9780712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.864+0000 7fb97d85e640 1 -- 172.21.15.28:0/4087198936 shutdown_connections 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.864+0000 7fb97d85e640 1 -- 172.21.15.28:0/4087198936 wait complete. 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.865+0000 7fb97d85e640 1 Processor -- start 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.866+0000 7fb97d85e640 1 -- start start 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.866+0000 7fb97d85e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810a650 0x7fb9781a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.867+0000 7fb97d85e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.867+0000 7fb97d85e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 0x7fb9781a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.867+0000 7fb97d85e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb97811a110 con 0x7fb97810d620 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.867+0000 7fb97d85e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb978119f90 con 0x7fb97810ccd0 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.867+0000 7fb97d85e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb97811a290 con 0x7fb97810a650 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.868+0000 7fb9767fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.868+0000 7fb976ffd640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810a650 0x7fb9781a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.868+0000 7fb9767fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47228/0 (socket says 172.21.15.28:47228) 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.868+0000 7fb9767fc640 1 -- 172.21.15.28:0/1336249662 learned_addr learned my addr 172.21.15.28:0/1336249662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.868+0000 7fb9777fe640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 0x7fb9781a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9777fe640 1 -- 172.21.15.28:0/1336249662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810a650 msgr2=0x7fb9781a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9777fe640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810a650 0x7fb9781a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9777fe640 1 -- 172.21.15.28:0/1336249662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 msgr2=0x7fb9781a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9777fe640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9777fe640 1 -- 172.21.15.28:0/1336249662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9781a6ab0 con 0x7fb97810d620 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.869+0000 7fb9767fc640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.871+0000 7fb9777fe640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 0x7fb9781a6380 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb97000ac30 tx=0x7fb970007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.872+0000 7fb97c85c640 1 -- 172.21.15.28:0/1336249662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970019020 con 0x7fb97810d620 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.872+0000 7fb97c85c640 1 -- 172.21.15.28:0/1336249662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb970004d20 con 0x7fb97810d620 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.872+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9781a6da0 con 0x7fb97810d620 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.872+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9781ad500 con 0x7fb97810d620 2024-09-21T05:53:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.872+0000 7fb97c85c640 1 -- 172.21.15.28:0/1336249662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970015660 con 0x7fb97810d620 2024-09-21T05:53:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.878+0000 7fb983dd2740 1 Processor -- start 2024-09-21T05:53:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 -- start start 2024-09-21T05:53:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6695b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62cd390 0x5614f66960d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614f660be50 con 0x5614f62cd390 2024-09-21T05:53:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5614f65fb280 con 0x5614f62ee070 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.881+0000 7fb983dd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5614f65ff090 con 0x5614f62c51b0 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.882+0000 7fb9767fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.882+0000 7fb9767fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47238/0 (socket says 172.21.15.28:47238) 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.882+0000 7fb9767fc640 1 -- 172.21.15.28:0/2317204641 learned_addr learned my addr 172.21.15.28:0/2317204641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.882+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62cd390 0x5614f66960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.882+0000 7fb97c85c640 1 -- 172.21.15.28:0/1336249662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb97000c030 con 0x7fb97810d620 2024-09-21T05:53:38.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.883+0000 7fb9767fc640 1 -- 172.21.15.28:0/2317204641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 msgr2=0x5614f6695b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.883+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6695b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.883+0000 7fb9767fc640 1 -- 172.21.15.28:0/2317204641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62cd390 msgr2=0x5614f66960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.883+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62cd390 0x5614f66960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.883+0000 7fb9767fc640 1 -- 172.21.15.28:0/2317204641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f6359650 con 0x5614f62ee070 2024-09-21T05:53:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.884+0000 7fb97c85c640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 0x7fb9600b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.885+0000 7fb97c85c640 1 -- 172.21.15.28:0/1336249662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb9700e72b0 con 0x7fb97810d620 2024-09-21T05:53:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.885+0000 7fb976ffd640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 0x7fb9600b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.885+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb968005f60 tx=0x7fb96800bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13aee914b1d574ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.889+0000 7fb95dffb640 1 -- 172.21.15.28:0/2317204641 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968012cd0 con 0x5614f62ee070 2024-09-21T05:53:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.889+0000 7fb95dffb640 1 -- 172.21.15.28:0/2317204641 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968010c00 con 0x5614f62ee070 2024-09-21T05:53:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.890+0000 7fb95dffb640 1 -- 172.21.15.28:0/2317204641 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968003b20 con 0x5614f62ee070 2024-09-21T05:53:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 -- 172.21.15.28:0/2317204641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 msgr2=0x5614f6695650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb968005f60 tx=0x7fb96800bc10 comp rx=0 tx=0).stop 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 -- 172.21.15.28:0/2317204641 shutdown_connections 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62cd390 0x5614f66960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6695b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2317204641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ee070 0x5614f6695650 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb968005f60 tx=0x7fb96800bc10 comp rx=0 tx=0).stop 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 -- 172.21.15.28:0/2317204641 >> 172.21.15.28:0/2317204641 conn(0x5614f62e2460 msgr2=0x5614f6693540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:38.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 -- 172.21.15.28:0/2317204641 shutdown_connections 2024-09-21T05:53:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.891+0000 7fb983dd2740 1 -- 172.21.15.28:0/2317204641 wait complete. 2024-09-21T05:53:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.892+0000 7fb983dd2740 1 -- start start 2024-09-21T05:53:38.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.892+0000 7fb983dd2740 1 Processor -- start 2024-09-21T05:53:38.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62c51b0 0x5614f66a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614f645c700 con 0x5614f62ac9f0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5614f6596110 con 0x5614f62c51b0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5614f66a3e80 con 0x5614f62cd390 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb983dd2740 10 client.0 _ll_register_callbacks cb 0x5614f65af990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.893+0000 7fb9777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.894+0000 7fb9777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50834/0 (socket says 172.21.15.28:50834) 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.894+0000 7fb9777fe640 1 -- 172.21.15.28:0/2336065656 learned_addr learned my addr 172.21.15.28:0/2336065656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.894+0000 7fb976ffd640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 0x7fb9600b8d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0084c0 tx=0x7fb96c005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.894+0000 7fb976ffd640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62c51b0 0x5614f66a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.894+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.896+0000 7fb9777fe640 1 -- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 msgr2=0x5614f66a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.896+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.896+0000 7fb9777fe640 1 -- 172.21.15.28:0/2336065656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62c51b0 msgr2=0x5614f66a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.896+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62c51b0 0x5614f66a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.896+0000 7fb9777fe640 1 -- 172.21.15.28:0/2336065656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f65c3c40 con 0x5614f62ac9f0 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.897+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ace50 tx=0x7fb9700acfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90443a8c9db131a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.897+0000 7fb956ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:38.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.897+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9700983c0 con 0x5614f62ac9f0 2024-09-21T05:53:38.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.897+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb968010c00 con 0x5614f62ac9f0 2024-09-21T05:53:38.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.897+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb9700ac3b0 con 0x5614f62ac9f0 2024-09-21T05:53:38.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.899+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb9700ad850 con 0x5614f62ac9f0 2024-09-21T05:53:38.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.899+0000 7fb983dd2740 10 client.17112 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:38.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.899+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5614f66afc90 con 0x5614f62ac9f0 2024-09-21T05:53:38.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.899+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb968010c00 con 0x5614f62ac9f0 2024-09-21T05:53:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fb9700c5070 con 0x5614f62ac9f0 2024-09-21T05:53:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 10 client.17112 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 20 client.17112 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:38.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f6281da0 0x5614f66b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 4 client.17112 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:38.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.901+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614f660dda0 con 0x5614f6281da0 2024-09-21T05:53:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.904+0000 7fb976ffd640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f6281da0 0x5614f66b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.913+0000 7fb956ffd640 10 client.17112 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.913+0000 7fb976ffd640 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f6281da0 0x5614f66b3720 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bffa954238d2fed6 server_cookie=a3ac9f82a588984d in_seq=0 out_seq=0 2024-09-21T05:53:38.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.914+0000 7fb956ffd640 1 -- 172.21.15.28:0/2336065656 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5614f660dda0 con 0x5614f6281da0 2024-09-21T05:53:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.914+0000 7fb956ffd640 10 client.17112 handle_command_reply: tid=0 2024-09-21T05:53:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.914+0000 7fb983dd2740 1 client.17112 shutdown 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.916+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 msgr2=0x5614f66ad250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.916+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ace50 tx=0x7fb9700acfe0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 shutdown_connections 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f6281da0 0x5614f66b3720 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f66ad250 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2336065656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62c51b0 0x5614f66a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 >> 172.21.15.28:0/2336065656 conn(0x5614f62e2460 msgr2=0x5614f66987e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.917+0000 7fb956ffd640 0 client.17112 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.920+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 shutdown_connections 2024-09-21T05:53:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.920+0000 7fb983dd2740 1 -- 172.21.15.28:0/2336065656 wait complete. 2024-09-21T05:53:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.920+0000 7fb983dd2740 20 client.17112 trim_cache size 0 max 16384 2024-09-21T05:53:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.956+0000 7fb983dd2740 1 Processor -- start 2024-09-21T05:53:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.956+0000 7fb983dd2740 1 -- start start 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f6696d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6692b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb983dd2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb96400e730 con 0x5614f62ac9f0 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb983dd2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9640102d0 con 0x5614f62cd390 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb9777fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb9777fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47248/0 (socket says 172.21.15.28:47248) 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.957+0000 7fb9777fe640 1 -- 172.21.15.28:0/2345389612 learned_addr learned my addr 172.21.15.28:0/2345389612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb96400ee00 con 0x5614f62c51b0 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9767fc640 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6692b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9777fe640 1 -- 172.21.15.28:0/2345389612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 msgr2=0x5614f6692b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6692b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9777fe640 1 -- 172.21.15.28:0/2345389612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 msgr2=0x5614f6696d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f6696d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.958+0000 7fb9777fe640 1 -- 172.21.15.28:0/2345389612 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f6697ee0 con 0x5614f62cd390 2024-09-21T05:53:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.962+0000 7fb9777fe640 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb9700aa3c0 tx=0x7fb9700ad050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23071782c16372 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.971+0000 7fb95d7fa640 1 -- 172.21.15.28:0/2345389612 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9700ac5b0 con 0x5614f62cd390 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.971+0000 7fb95d7fa640 1 -- 172.21.15.28:0/2345389612 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9700e7040 con 0x5614f62cd390 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.971+0000 7fb95d7fa640 1 -- 172.21.15.28:0/2345389612 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9700a8d00 con 0x5614f62cd390 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.972+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 msgr2=0x5614f66930b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb9700aa3c0 tx=0x7fb9700ad050 comp rx=0 tx=0).stop 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 shutdown_connections 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62cd390 0x5614f66930b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62c51b0 0x5614f6692b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 --2- 172.21.15.28:0/2345389612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62ac9f0 0x5614f6696d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.973+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 >> 172.21.15.28:0/2345389612 conn(0x5614f62e2460 msgr2=0x5614f6696680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.975+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 shutdown_connections 2024-09-21T05:53:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.975+0000 7fb983dd2740 1 -- 172.21.15.28:0/2345389612 wait complete. 2024-09-21T05:53:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.975+0000 7fb983dd2740 1 -- start start 2024-09-21T05:53:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.976+0000 7fb983dd2740 1 Processor -- start 2024-09-21T05:53:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62c51b0 0x5614f66a5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb976ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb976ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47254/0 (socket says 172.21.15.28:47254) 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb976ffd640 1 -- 172.21.15.28:0/389398925 learned_addr learned my addr 172.21.15.28:0/389398925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb96400fc00 con 0x5614f62c51b0 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb9777fe640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb96400f4d0 con 0x5614f62ac9f0 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.977+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb96400c7c0 con 0x5614f62cd390 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.978+0000 7fb983dd2740 10 client.0 _ll_register_callbacks cb 0x7fb97000aea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.979+0000 7fb9767fc640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62c51b0 0x5614f66a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 -- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 msgr2=0x5614f66a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 -- 172.21.15.28:0/389398925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62c51b0 msgr2=0x5614f66a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62c51b0 0x5614f66a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f669b980 con 0x5614f62ac9f0 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb976ffd640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5614f66942d0 tx=0x7fb96c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e92c099aefc4498 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.980+0000 7fb956ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c0035f0 con 0x5614f62ac9f0 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9700e7040 con 0x5614f62ac9f0 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fb96c003790 con 0x5614f62ac9f0 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb96c003990 con 0x5614f62ac9f0 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb983dd2740 10 client.26699 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.981+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5614f671e290 con 0x5614f62ac9f0 2024-09-21T05:53:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9700e7040 con 0x5614f62ac9f0 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fb96c041450 con 0x5614f62ac9f0 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 10 client.26699 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 20 client.26699 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f671ea70 0x5614f6720e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 4 client.26699 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5614f6609850 con 0x5614f671ea70 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.982+0000 7fb9767fc640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f671ea70 0x5614f6720e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.983+0000 7fb956ffd640 10 client.26699 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.984+0000 7fb9767fc640 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f671ea70 0x5614f6720e90 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37249783aa1d1405 server_cookie=2d536c78ca9fb61f in_seq=0 out_seq=0 2024-09-21T05:53:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.992+0000 7fb956ffd640 1 -- 172.21.15.28:0/389398925 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3134 (crc 0 0 0) 0x5614f6609850 con 0x5614f671ea70 2024-09-21T05:53:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.992+0000 7fb956ffd640 10 client.26699 handle_command_reply: tid=0 2024-09-21T05:53:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.992+0000 7fb983dd2740 1 client.26699 shutdown 2024-09-21T05:53:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.992+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 msgr2=0x5614f66a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.992+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5614f66942d0 tx=0x7fb96c000bc0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 shutdown_connections 2024-09-21T05:53:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5614f671ea70 0x5614f6720e90 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5614f62cd390 0x5614f66a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614f62c51b0 0x5614f66a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 --2- 172.21.15.28:0/389398925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5614f62ac9f0 0x5614f66a5200 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 >> 172.21.15.28:0/389398925 conn(0x5614f62e2460 msgr2=0x5614f6697040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb956ffd640 0 client.26699 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.994+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 shutdown_connections 2024-09-21T05:53:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.996+0000 7fb983dd2740 1 -- 172.21.15.28:0/389398925 wait complete. 2024-09-21T05:53:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:38.998+0000 7fb983dd2740 20 client.26699 trim_cache size 0 max 16384 2024-09-21T05:53:39.001 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:53:39.001 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (111 inodes in the stack)", 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "b3760c46-b2b6-415c-99f0-96d4557921a0", 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.002 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:53:39.003 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:53:39.006 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:53:39.007 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:53:39.008 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.003+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 msgr2=0x7fb9600b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.003+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 0x7fb9600b8d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0084c0 tx=0x7fb96c005fd0 comp rx=0 tx=0).stop 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.003+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 msgr2=0x7fb9781a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.003+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 0x7fb9781a6380 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb97000ac30 tx=0x7fb970007590 comp rx=0 tx=0).stop 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 shutdown_connections 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb9600b6880 0x7fb9600b8d90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb97810d620 0x7fb9781a6380 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97810ccd0 0x7fb9781a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 --2- 172.21.15.28:0/1336249662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97810a650 0x7fb9781a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 >> 172.21.15.28:0/1336249662 conn(0x7fb978070ec0 msgr2=0x7fb97810ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.004+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 shutdown_connections 2024-09-21T05:53:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:53:39.005+0000 7fb97d85e640 1 -- 172.21.15.28:0/1336249662 wait complete. 2024-09-21T05:53:40.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:40 smithi028 ceph-mon[31227]: pgmap v524: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 3.50k op/s 2024-09-21T05:53:40.317 INFO:tasks.cephfs.filesystem:scrub status for tag:b3760c46-b2b6-415c-99f0-96d4557921a0 - {'path': '/', 'tag': 'b3760c46-b2b6-415c-99f0-96d4557921a0', 'options': 'recursive,force'} 2024-09-21T05:53:40.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:40 smithi129 ceph-mon[34885]: pgmap v524: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 3.50k op/s 2024-09-21T05:53:40.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:40 smithi181 ceph-mon[34819]: pgmap v524: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 3.50k op/s 2024-09-21T05:53:42.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:42 smithi129 ceph-mon[34885]: pgmap v525: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-09-21T05:53:42.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:42 smithi181 ceph-mon[34819]: pgmap v525: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-09-21T05:53:42.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:42 smithi028 ceph-mon[31227]: pgmap v525: 97 pgs: 97 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 4.40k op/s 2024-09-21T05:53:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.025+0000 7f9380ca2640 1 -- 172.21.15.28:0/4181358509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c075dd0 msgr2=0x7f937c0761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.025+0000 7f9380ca2640 1 --2- 172.21.15.28:0/4181358509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c075dd0 0x7f937c0761e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9364009900 tx=0x7f936402a440 comp rx=0 tx=0).stop 2024-09-21T05:53:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.033+0000 7f9380ca2640 1 -- 172.21.15.28:0/4181358509 shutdown_connections 2024-09-21T05:53:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.033+0000 7f9380ca2640 1 --2- 172.21.15.28:0/4181358509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c1143d0 0x7f937c1168a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.033+0000 7f9380ca2640 1 --2- 172.21.15.28:0/4181358509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c075dd0 0x7f937c0761e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.033+0000 7f9380ca2640 1 --2- 172.21.15.28:0/4181358509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c077390 0x7f937c0757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.033+0000 7f9380ca2640 1 -- 172.21.15.28:0/4181358509 >> 172.21.15.28:0/4181358509 conn(0x7f937c0ff510 msgr2=0x7f937c101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.039+0000 7f9380ca2640 1 -- 172.21.15.28:0/4181358509 shutdown_connections 2024-09-21T05:53:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.039+0000 7f9380ca2640 1 -- 172.21.15.28:0/4181358509 wait complete. 2024-09-21T05:53:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.041+0000 7f9380ca2640 1 Processor -- start 2024-09-21T05:53:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 -- start start 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 0x7f937c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 0x7f937c06f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c1143d0 0x7f937c06f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f937c06d6e0 con 0x7f937c1143d0 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f937c06d560 con 0x7f937c077390 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.044+0000 7f9380ca2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f937c06d860 con 0x7f937c075dd0 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.045+0000 7f9373fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 0x7f937c06f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.045+0000 7f9373fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 0x7f937c06f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47262/0 (socket says 172.21.15.28:47262) 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.045+0000 7f9373fff640 1 -- 172.21.15.28:0/2826525690 learned_addr learned my addr 172.21.15.28:0/2826525690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.045+0000 7f937a59c640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 0x7f937c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.046+0000 7f937a59c640 1 -- 172.21.15.28:0/2826525690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 msgr2=0x7f937c06f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.046+0000 7f937a59c640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 0x7f937c06f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.046+0000 7f937a59c640 1 -- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c1143d0 msgr2=0x7f937c06f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.046+0000 7f937a59c640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c1143d0 0x7f937c06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.046+0000 7f937a59c640 1 -- 172.21.15.28:0/2826525690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937c107ac0 con 0x7f937c075dd0 2024-09-21T05:53:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.049+0000 7f937a59c640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 0x7f937c075050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d6d0 tx=0x7f936c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.049+0000 7f93737fe640 1 -- 172.21.15.28:0/2826525690 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c017070 con 0x7f937c075dd0 2024-09-21T05:53:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.050+0000 7f93737fe640 1 -- 172.21.15.28:0/2826525690 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f936c009820 con 0x7f937c075dd0 2024-09-21T05:53:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.050+0000 7f93737fe640 1 -- 172.21.15.28:0/2826525690 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c004c40 con 0x7f937c075dd0 2024-09-21T05:53:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.051+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f937c107d50 con 0x7f937c075dd0 2024-09-21T05:53:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.051+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f937c108250 con 0x7f937c075dd0 2024-09-21T05:53:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.057+0000 7f93737fe640 1 -- 172.21.15.28:0/2826525690 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f936c005220 con 0x7f937c075dd0 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.058+0000 7f9387c16740 1 Processor -- start 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 -- start start 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 0x55884bf319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf31ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55884bea66f0 con 0x55884bb8c1b0 2024-09-21T05:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55884be95b20 con 0x55884bb6b4d0 2024-09-21T05:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f9387c16740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55884be99930 con 0x55884bb632f0 2024-09-21T05:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.063+0000 7f93737fe640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 0x7f93680b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.064+0000 7f93737fe640 1 -- 172.21.15.28:0/2826525690 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f936c0e7070 con 0x7f937c075dd0 2024-09-21T05:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.065+0000 7f937ad9d640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 0x7f93680b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.065+0000 7f937ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.065+0000 7f937ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47270/0 (socket says 172.21.15.28:47270) 2024-09-21T05:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.065+0000 7f937ad9d640 1 -- 172.21.15.28:0/1076220661 learned_addr learned my addr 172.21.15.28:0/1076220661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.066+0000 7f937ad9d640 1 -- 172.21.15.28:0/1076220661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 msgr2=0x55884bf31ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.066+0000 7f937ad9d640 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf31ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.067+0000 7f937ad9d640 1 -- 172.21.15.28:0/1076220661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 msgr2=0x55884bf319b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:53:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.067+0000 7f937a59c640 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 0x55884bf319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.067+0000 7f937ad9d640 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 0x55884bf319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.068+0000 7f937ad9d640 1 -- 172.21.15.28:0/1076220661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55884be5b530 con 0x55884bb6b4d0 2024-09-21T05:53:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.068+0000 7f937ad9d640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 0x7f93680b8d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9374012a80 tx=0x7f9374009cd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.069+0000 7f937a59c640 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 0x55884bf319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:53:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.070+0000 7f937ad9d640 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f937400fc60 tx=0x7f9374019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f857697c8446183 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.071+0000 7f9370ff9640 1 -- 172.21.15.28:0/1076220661 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374022070 con 0x55884bb6b4d0 2024-09-21T05:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.072+0000 7f9370ff9640 1 -- 172.21.15.28:0/1076220661 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f937400dbe0 con 0x55884bb6b4d0 2024-09-21T05:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.072+0000 7f9370ff9640 1 -- 172.21.15.28:0/1076220661 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374005490 con 0x55884bb6b4d0 2024-09-21T05:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.074+0000 7f9387c16740 1 -- 172.21.15.28:0/1076220661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 msgr2=0x55884bf32430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.074+0000 7f9387c16740 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f937400fc60 tx=0x7f9374019000 comp rx=0 tx=0).stop 2024-09-21T05:53:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.079+0000 7f9387c16740 1 -- 172.21.15.28:0/1076220661 shutdown_connections 2024-09-21T05:53:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.079+0000 7f9387c16740 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb6b4d0 0x55884bf32430 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.079+0000 7f9387c16740 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf31ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.079+0000 7f9387c16740 1 --2- 172.21.15.28:0/1076220661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb8c1b0 0x55884bf319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.079+0000 7f9387c16740 1 -- 172.21.15.28:0/1076220661 >> 172.21.15.28:0/1076220661 conn(0x55884bb80300 msgr2=0x55884bf2f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:43.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.084+0000 7f9387c16740 1 -- 172.21.15.28:0/1076220661 shutdown_connections 2024-09-21T05:53:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.086+0000 7f9387c16740 1 -- 172.21.15.28:0/1076220661 wait complete. 2024-09-21T05:53:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.086+0000 7f9387c16740 1 -- start start 2024-09-21T05:53:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.086+0000 7f9387c16740 1 Processor -- start 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf49420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55884bdcdab0 con 0x55884bb6b4d0 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55884be3b730 con 0x55884bb8c1b0 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55884bf400d0 con 0x55884bb632f0 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f9387c16740 10 client.0 _ll_register_callbacks cb 0x55884be8f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f937ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf49420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f937a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f937a59c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:44234/0 (socket says 172.21.15.28:44234) 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.088+0000 7f937a59c640 1 -- 172.21.15.28:0/2880493291 learned_addr learned my addr 172.21.15.28:0/2880493291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.089+0000 7f937a59c640 1 -- 172.21.15.28:0/2880493291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 msgr2=0x55884bf49420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.089+0000 7f937a59c640 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf49420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.089+0000 7f937a59c640 1 -- 172.21.15.28:0/2880493291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 msgr2=0x55884bf3d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.089+0000 7f937a59c640 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.089+0000 7f937a59c640 1 -- 172.21.15.28:0/2880493291 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55884bbf7790 con 0x55884bb632f0 2024-09-21T05:53:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.090+0000 7f937a59c640 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f936c0aa0f0 tx=0x7f936c009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed588a426a5bc451 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.092+0000 7f9361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.092+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c020040 con 0x55884bb632f0 2024-09-21T05:53:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.092+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937400dbe0 con 0x55884bb632f0 2024-09-21T05:53:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.092+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f936c0e4a00 con 0x55884bb632f0 2024-09-21T05:53:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.093+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f936c0e4c00 con 0x55884bb632f0 2024-09-21T05:53:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.094+0000 7f9387c16740 10 client.26704 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.094+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55884bc70620 con 0x55884bb632f0 2024-09-21T05:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.097+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f937400dbe0 con 0x55884bb632f0 2024-09-21T05:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.097+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f936c0b4ce0 con 0x55884bb632f0 2024-09-21T05:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 10 client.26704 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 10 client.26704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 20 client.26704 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bb4ad80 0x55884bf4f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 4 client.26704 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55884bea8640 con 0x55884bb4ad80 2024-09-21T05:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.098+0000 7f9373fff640 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bb4ad80 0x55884bf4f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.109+0000 7f9373fff640 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bb4ad80 0x55884bf4f210 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf9f78b49f05d63 server_cookie=b5b2b0cd76230b78 in_seq=0 out_seq=0 2024-09-21T05:53:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.110+0000 7f9361ffb640 10 client.26704 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:43.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.115+0000 7f9361ffb640 1 -- 172.21.15.28:0/2880493291 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55884bea8640 con 0x55884bb4ad80 2024-09-21T05:53:43.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.115+0000 7f9361ffb640 10 client.26704 handle_command_reply: tid=0 2024-09-21T05:53:43.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.116+0000 7f9387c16740 1 client.26704 shutdown 2024-09-21T05:53:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.121+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 msgr2=0x55884bf3d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.121+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f936c0aa0f0 tx=0x7f936c009f00 comp rx=0 tx=0).stop 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 shutdown_connections 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bb4ad80 0x55884bf4f210 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf49420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 --2- 172.21.15.28:0/2880493291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb632f0 0x55884bf3d0f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 >> 172.21.15.28:0/2880493291 conn(0x55884bb80300 msgr2=0x55884bf329e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.125+0000 7f9361ffb640 0 client.26704 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.129+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 shutdown_connections 2024-09-21T05:53:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.130+0000 7f9387c16740 1 -- 172.21.15.28:0/2880493291 wait complete. 2024-09-21T05:53:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.130+0000 7f9387c16740 20 client.26704 trim_cache size 0 max 16384 2024-09-21T05:53:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.161+0000 7f9387c16740 1 Processor -- start 2024-09-21T05:53:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.161+0000 7f9387c16740 1 -- start start 2024-09-21T05:53:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.162+0000 7f9387c16740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 0x55884bf387f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.162+0000 7f9387c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.162+0000 7f9387c16740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb8c1b0 0x55884bf330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.162+0000 7f9373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.162+0000 7f9373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50892/0 (socket says 172.21.15.28:50892) 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.163+0000 7f9373fff640 1 -- 172.21.15.28:0/2061306501 learned_addr learned my addr 172.21.15.28:0/2061306501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.163+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f937c1c6e10 con 0x55884bb6b4d0 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.163+0000 7f937ad9d640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb8c1b0 0x55884bf330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.163+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f937c1c89b0 con 0x55884bb632f0 2024-09-21T05:53:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.163+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f937c1c74e0 con 0x55884bb8c1b0 2024-09-21T05:53:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.164+0000 7f937a59c640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 0x55884bf387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f9373fff640 1 -- 172.21.15.28:0/2061306501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb8c1b0 msgr2=0x55884bf330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f9373fff640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb8c1b0 0x55884bf330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f9373fff640 1 -- 172.21.15.28:0/2061306501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 msgr2=0x55884bf387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f9373fff640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 0x55884bf387f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f9373fff640 1 -- 172.21.15.28:0/2061306501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55884bf32790 con 0x55884bb6b4d0 2024-09-21T05:53:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.166+0000 7f937a59c640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 0x55884bf387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.169+0000 7f9373fff640 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f936402aa00 tx=0x7f93640036b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d243e3538666d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.171+0000 7f9363fff640 1 -- 172.21.15.28:0/2061306501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364004940 con 0x55884bb6b4d0 2024-09-21T05:53:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.171+0000 7f9363fff640 1 -- 172.21.15.28:0/2061306501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93640318f0 con 0x55884bb6b4d0 2024-09-21T05:53:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.171+0000 7f9363fff640 1 -- 172.21.15.28:0/2061306501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936403d900 con 0x55884bb6b4d0 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 msgr2=0x55884bf3efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f936402aa00 tx=0x7f93640036b0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 shutdown_connections 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb8c1b0 0x55884bf330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb6b4d0 0x55884bf3efd0 secure :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f936402aa00 tx=0x7f93640036b0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 --2- 172.21.15.28:0/2061306501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb632f0 0x55884bf387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.175+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 >> 172.21.15.28:0/2061306501 conn(0x55884bb80300 msgr2=0x55884bf329e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.184+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 shutdown_connections 2024-09-21T05:53:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.185+0000 7f9387c16740 1 -- 172.21.15.28:0/2061306501 wait complete. 2024-09-21T05:53:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.185+0000 7f9387c16740 1 -- start start 2024-09-21T05:53:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 Processor -- start 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb632f0 0x55884bf45930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 0x55884bf41310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f937c1c82e0 con 0x55884bb632f0 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f937c1c7bb0 con 0x55884bb8c1b0 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f937c10ec20 con 0x55884bb201c0 2024-09-21T05:53:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.186+0000 7f9387c16740 10 client.0 _ll_register_callbacks cb 0x7f936c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.187+0000 7f9373fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.187+0000 7f9373fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:47296/0 (socket says 172.21.15.28:47296) 2024-09-21T05:53:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.187+0000 7f9373fff640 1 -- 172.21.15.28:0/3358676484 learned_addr learned my addr 172.21.15.28:0/3358676484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.189+0000 7f937ad9d640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 0x55884bf41310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937a59c640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb632f0 0x55884bf45930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937ad9d640 1 -- 172.21.15.28:0/3358676484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 msgr2=0x55884bf40dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937ad9d640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937ad9d640 1 -- 172.21.15.28:0/3358676484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb632f0 msgr2=0x55884bf45930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937ad9d640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb632f0 0x55884bf45930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.190+0000 7f937ad9d640 1 -- 172.21.15.28:0/3358676484 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55884bf41880 con 0x55884bb201c0 2024-09-21T05:53:43.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.191+0000 7f9373fff640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:53:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.195+0000 7f937ad9d640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 0x55884bf41310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f937400f730 tx=0x7f937400ff80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd37cb5cff34d349 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.197+0000 7f9361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:53:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.199+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374022040 con 0x55884bb201c0 2024-09-21T05:53:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.199+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7f937401e440 con 0x55884bb201c0 2024-09-21T05:53:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.201+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93640318f0 con 0x55884bb201c0 2024-09-21T05:53:43.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.206+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9374012450 con 0x55884bb201c0 2024-09-21T05:53:43.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.206+0000 7f9387c16740 10 client.26710 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:43.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.206+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55884bfb9940 con 0x55884bb201c0 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.211+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93640318f0 con 0x55884bb201c0 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.211+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7f9374005020 con 0x55884bb201c0 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 10 client.26710 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 10 client.26710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 20 client.26710 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bfb9c80 0x55884bfbc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 4 client.26710 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:53:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.212+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55884bea40f0 con 0x55884bfb9c80 2024-09-21T05:53:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.217+0000 7f937a59c640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bfb9c80 0x55884bfbc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.222+0000 7f937a59c640 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bfb9c80 0x55884bfbc0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3e2a5d49a630658 server_cookie=1251e30832d90cc1 in_seq=0 out_seq=0 2024-09-21T05:53:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:43.223+0000 7f9361ffb640 10 client.26710 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9361ffb640 1 -- 172.21.15.28:0/3358676484 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55884bea40f0 con 0x55884bfb9c80 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9361ffb640 10 client.26710 handle_command_reply: tid=0 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 client.26710 shutdown 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 msgr2=0x55884bf41310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 0x55884bf41310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f937400f730 tx=0x7f937400ff80 comp rx=0 tx=0).stop 2024-09-21T05:53:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 shutdown_connections 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55884bfb9c80 0x55884bfbc0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55884bb201c0 0x55884bf41310 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55884bb8c1b0 0x55884bf40dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 --2- 172.21.15.28:0/3358676484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55884bb632f0 0x55884bf45930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.221+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 >> 172.21.15.28:0/3358676484 conn(0x55884bb80300 msgr2=0x55884bf333a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.222+0000 7f9361ffb640 0 client.26710 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.222+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 shutdown_connections 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.222+0000 7f9387c16740 1 -- 172.21.15.28:0/3358676484 wait complete. 2024-09-21T05:53:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.222+0000 7f9387c16740 20 client.26710 trim_cache size 0 max 16384 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 msgr2=0x7f93680b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 0x7f93680b8d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9374012a80 tx=0x7f9374009cd0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 msgr2=0x7f937c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 0x7f937c075050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d6d0 tx=0x7f936c00db90 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 shutdown_connections 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f93680b6880 0x7f93680b8d90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f937c1143d0 0x7f937c06f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937c077390 0x7f937c06f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 --2- 172.21.15.28:0/2826525690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f937c075dd0 0x7f937c075050 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 >> 172.21.15.28:0/2826525690 conn(0x7f937c0ff510 msgr2=0x7f937c1102d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 shutdown_connections 2024-09-21T05:53:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:44.223+0000 7f9380ca2640 1 -- 172.21.15.28:0/2826525690 wait complete. 2024-09-21T05:53:44.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:44 smithi129 ceph-mon[34885]: pgmap v526: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 6.27k op/s 2024-09-21T05:53:44.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:44 smithi181 ceph-mon[34819]: pgmap v526: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 6.27k op/s 2024-09-21T05:53:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:44 smithi028 ceph-mon[31227]: pgmap v526: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 6.27k op/s 2024-09-21T05:53:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:46 smithi129 ceph-mon[34885]: pgmap v527: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 7.83k op/s 2024-09-21T05:53:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:46 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:46 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:46 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:46 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:46 smithi181 ceph-mon[34819]: pgmap v527: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 7.83k op/s 2024-09-21T05:53:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:46 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:46 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:46 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:46 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:46 smithi028 ceph-mon[31227]: pgmap v527: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 7.83k op/s 2024-09-21T05:53:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:46 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:46 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:46 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:46 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:47.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:47 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:47.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:47 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:47.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:47 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:53:48.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:48 smithi129 ceph-mon[34885]: pgmap v528: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 8.12k op/s 2024-09-21T05:53:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:48 smithi181 ceph-mon[34819]: pgmap v528: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 8.12k op/s 2024-09-21T05:53:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:48 smithi028 ceph-mon[31227]: pgmap v528: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 36 KiB/s rd, 34 MiB/s wr, 8.12k op/s 2024-09-21T05:53:50.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:50 smithi129 ceph-mon[34885]: pgmap v529: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 78 KiB/s rd, 39 MiB/s wr, 9.15k op/s 2024-09-21T05:53:50.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:50 smithi181 ceph-mon[34819]: pgmap v529: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 78 KiB/s rd, 39 MiB/s wr, 9.15k op/s 2024-09-21T05:53:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:50 smithi028 ceph-mon[31227]: pgmap v529: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 78 KiB/s rd, 39 MiB/s wr, 9.15k op/s 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: pgmap v530: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 80 KiB/s rd, 33 MiB/s wr, 7.88k op/s 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:52.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:52 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: pgmap v530: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 80 KiB/s rd, 33 MiB/s wr, 7.88k op/s 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:52 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: pgmap v530: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 80 KiB/s rd, 33 MiB/s wr, 7.88k op/s 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:52 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:53:53.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:53 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:53:53.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:53 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:53:53.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:53 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:53:54.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: pgmap v531: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 83 KiB/s rd, 29 MiB/s wr, 6.83k op/s 2024-09-21T05:53:54.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi181 to 176.0M 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi129 to 176.0M 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Unable to set osd_memory_target on smithi181 to 184558796: error parsing value: Value '184558796' is below minimum 939524096 2024-09-21T05:53:54.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Unable to set osd_memory_target on smithi129 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-21T05:53:54.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: Unable to set osd_memory_target on smithi028 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T05:53:54.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:53:54.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:53:54.366 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:53:54.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: pgmap v531: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 83 KiB/s rd, 29 MiB/s wr, 6.83k op/s 2024-09-21T05:53:54.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi181 to 176.0M 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi129 to 176.0M 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Unable to set osd_memory_target on smithi181 to 184558796: error parsing value: Value '184558796' is below minimum 939524096 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Unable to set osd_memory_target on smithi129 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-21T05:53:54.445 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: Unable to set osd_memory_target on smithi028 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T05:53:54.446 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:53:54.446 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:53:54.446 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:53:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: pgmap v531: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 83 KiB/s rd, 29 MiB/s wr, 6.83k op/s 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi181 to 176.0M 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi129 to 176.0M 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Unable to set osd_memory_target on smithi181 to 184558796: error parsing value: Value '184558796' is below minimum 939524096 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Unable to set osd_memory_target on smithi129 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: Unable to set osd_memory_target on smithi028 to 185687756: error parsing value: Value '185687756' is below minimum 939524096 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:53:54.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:53:56.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:56 smithi129 ceph-mon[34885]: pgmap v532: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 74 KiB/s rd, 20 MiB/s wr, 4.71k op/s 2024-09-21T05:53:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:56 smithi181 ceph-mon[34819]: pgmap v532: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 74 KiB/s rd, 20 MiB/s wr, 4.71k op/s 2024-09-21T05:53:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:56 smithi028 ceph-mon[31227]: pgmap v532: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 74 KiB/s rd, 20 MiB/s wr, 4.71k op/s 2024-09-21T05:53:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 -- 172.21.15.28:0/3878898550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434105250 msgr2=0x7fd43410d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3878898550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434105250 0x7fd43410d790 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd428009900 tx=0x7fd42802a410 comp rx=0 tx=0).stop 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 -- 172.21.15.28:0/3878898550 shutdown_connections 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3878898550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43410dd60 0x7fd434110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3878898550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434105250 0x7fd43410d790 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3878898550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434104900 0x7fd434104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 -- 172.21.15.28:0/3878898550 >> 172.21.15.28:0/3878898550 conn(0x7fd43407a740 msgr2=0x7fd43407ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 -- 172.21.15.28:0/3878898550 shutdown_connections 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 -- 172.21.15.28:0/3878898550 wait complete. 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.337+0000 7fd4396f2640 1 Processor -- start 2024-09-21T05:53:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 -- start start 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434105250 0x7fd43419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43410dd60 0x7fd4341a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341139c0 con 0x7fd434104900 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd434113840 con 0x7fd43410dd60 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4396f2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd434113b40 con 0x7fd434105250 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59656/0 (socket says 172.21.15.28:59656) 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 -- 172.21.15.28:0/3919435405 learned_addr learned my addr 172.21.15.28:0/3919435405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4337fe640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43410dd60 0x7fd4341a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd4327fc640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434105250 0x7fd43419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 -- 172.21.15.28:0/3919435405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434105250 msgr2=0x7fd43419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434105250 0x7fd43419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 -- 172.21.15.28:0/3919435405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43410dd60 msgr2=0x7fd4341a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43410dd60 0x7fd4341a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.338+0000 7fd432ffd640 1 -- 172.21.15.28:0/3919435405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4341a2530 con 0x7fd434104900 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd432ffd640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd42400d950 tx=0x7fd42400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd41bfff640 1 -- 172.21.15.28:0/3919435405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424017030 con 0x7fd434104900 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd41bfff640 1 -- 172.21.15.28:0/3919435405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd424004020 con 0x7fd434104900 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4341a2820 con 0x7fd434104900 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd41bfff640 1 -- 172.21.15.28:0/3919435405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4240047a0 con 0x7fd434104900 2024-09-21T05:53:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.339+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4341a8fb0 con 0x7fd434104900 2024-09-21T05:53:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 Processor -- start 2024-09-21T05:53:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 -- start start 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5603524586f0 con 0x56035213e1b0 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560352447b20 con 0x5603521152f0 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.342+0000 7fd44065f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56035244b930 con 0x56035211d4d0 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd41bfff640 1 -- 172.21.15.28:0/3919435405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd424009a30 con 0x7fd434104900 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59662/0 (socket says 172.21.15.28:59662) 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 -- 172.21.15.28:0/2176458198 learned_addr learned my addr 172.21.15.28:0/2176458198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4327fc640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd432ffd640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 -- 172.21.15.28:0/2176458198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 msgr2=0x5603524e4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524e4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 -- 172.21.15.28:0/2176458198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 msgr2=0x5603524e3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 -- 172.21.15.28:0/2176458198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56035240d530 con 0x56035213e1b0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd432ffd640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4327fc640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524e4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4337fe640 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009f50 tx=0x7fd42c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e88934536d990c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4197fa640 1 -- 172.21.15.28:0/2176458198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c004d20 con 0x56035213e1b0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4197fa640 1 -- 172.21.15.28:0/2176458198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd42c004050 con 0x56035213e1b0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.343+0000 7fd4197fa640 1 -- 172.21.15.28:0/2176458198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c005240 con 0x56035213e1b0 2024-09-21T05:53:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd41bfff640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 0x7fd41c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd432ffd640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 0x7fd41c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd41bfff640 1 -- 172.21.15.28:0/3919435405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd4240e70e0 con 0x7fd434104900 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 -- 172.21.15.28:0/2176458198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 msgr2=0x5603524e39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd432ffd640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 0x7fd41c0b8b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd4240e49a0 tx=0x7fd4240e9040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009f50 tx=0x7fd42c0118d0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 -- 172.21.15.28:0/2176458198 shutdown_connections 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524e4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 --2- 172.21.15.28:0/2176458198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524e39b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 -- 172.21.15.28:0/2176458198 >> 172.21.15.28:0/2176458198 conn(0x560352132300 msgr2=0x5603524e18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 -- 172.21.15.28:0/2176458198 shutdown_connections 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.344+0000 7fd44065f740 1 -- 172.21.15.28:0/2176458198 wait complete. 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.345+0000 7fd44065f740 1 -- start start 2024-09-21T05:53:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.345+0000 7fd44065f740 1 Processor -- start 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.345+0000 7fd44065f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035213e1b0 0x5603524fb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56035237fab0 con 0x56035211d4d0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5603523ed730 con 0x5603521152f0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5603524f2130 con 0x56035213e1b0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd44065f740 10 client.0 _ll_register_callbacks cb 0x5603524416b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42170/0 (socket says 172.21.15.28:42170) 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59664/0 (socket says 172.21.15.28:59664) 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4337fe640 1 -- 172.21.15.28:0/1357537969 learned_addr learned my addr 172.21.15.28:0/1357537969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035213e1b0 0x5603524fb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 -- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035213e1b0 msgr2=0x5603524fb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035213e1b0 0x5603524fb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 -- 172.21.15.28:0/1357537969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 msgr2=0x5603524ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 -- 172.21.15.28:0/1357537969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603521a9790 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd4327fc640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd42802a930 tx=0x7fd428032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74827aedf2a1128d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.346+0000 7fd416ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd428035040 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd42c004050 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd428031760 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd428004040 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 10 client.17136 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560352222620 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd42c004050 con 0x56035211d4d0 2024-09-21T05:53:57.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fd42804d020 con 0x56035211d4d0 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 10 client.17136 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 10 client.17136 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 20 client.17136 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520fcd80 0x5603525010c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 4 client.17136 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56035245a640 con 0x5603520fcd80 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.347+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520fcd80 0x5603525010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.348+0000 7fd416ff5640 10 client.17136 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.348+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520fcd80 0x5603525010c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9c2316f719435f server_cookie=94de5c45d180e27b in_seq=0 out_seq=0 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd416ff5640 1 -- 172.21.15.28:0/1357537969 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56035245a640 con 0x5603520fcd80 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd416ff5640 10 client.17136 handle_command_reply: tid=0 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 client.17136 shutdown 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 msgr2=0x5603524ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd42802a930 tx=0x7fd428032e90 comp rx=0 tx=0).stop 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 shutdown_connections 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520fcd80 0x5603525010c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035213e1b0 0x5603524fb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035211d4d0 0x5603524ef570 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 --2- 172.21.15.28:0/1357537969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 >> 172.21.15.28:0/1357537969 conn(0x560352132300 msgr2=0x5603524e49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd416ff5640 0 client.17136 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 shutdown_connections 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 1 -- 172.21.15.28:0/1357537969 wait complete. 2024-09-21T05:53:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.349+0000 7fd44065f740 20 client.17136 trim_cache size 0 max 16384 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.364+0000 7fd44065f740 1 Processor -- start 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.364+0000 7fd44065f740 1 -- start start 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524fb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524ec200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341b18f0 con 0x56035213e1b0 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341b3490 con 0x5603521152f0 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd44065f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341b1fc0 con 0x56035211d4d0 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42178/0 (socket says 172.21.15.28:42178) 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 -- 172.21.15.28:0/1508887956 learned_addr learned my addr 172.21.15.28:0/1508887956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4327fc640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524fb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 -- 172.21.15.28:0/1508887956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 msgr2=0x5603524fb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524fb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 -- 172.21.15.28:0/1508887956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 msgr2=0x5603524ec200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524ec200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4337fe640 1 -- 172.21.15.28:0/1508887956 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603524e67c0 con 0x5603521152f0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.365+0000 7fd4327fc640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.366+0000 7fd4337fe640 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd42c012fd0 tx=0x7fd42c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c89ecc98409c4a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.366+0000 7fd418ff9640 1 -- 172.21.15.28:0/1508887956 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c003d20 con 0x5603521152f0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.366+0000 7fd418ff9640 1 -- 172.21.15.28:0/1508887956 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd42c009b30 con 0x5603521152f0 2024-09-21T05:53:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.366+0000 7fd418ff9640 1 -- 172.21.15.28:0/1508887956 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0055a0 con 0x5603521152f0 2024-09-21T05:53:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- 172.21.15.28:0/1508887956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 msgr2=0x5603524e50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd42c012fd0 tx=0x7fd42c009500 comp rx=0 tx=0).stop 2024-09-21T05:53:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- 172.21.15.28:0/1508887956 shutdown_connections 2024-09-21T05:53:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x5603524ec200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 --2- 172.21.15.28:0/1508887956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524e50f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- 172.21.15.28:0/1508887956 >> 172.21.15.28:0/1508887956 conn(0x560352132300 msgr2=0x5603524e49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- 172.21.15.28:0/1508887956 shutdown_connections 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- 172.21.15.28:0/1508887956 wait complete. 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 -- start start 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.367+0000 7fd44065f740 1 Processor -- start 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524f5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x560352569a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341b2dc0 con 0x56035213e1b0 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341b2690 con 0x5603521152f0 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341af8d0 con 0x56035211d4d0 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd44065f740 10 client.0 _ll_register_callbacks cb 0x7fd428009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:53:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:42186/0 (socket says 172.21.15.28:42186) 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 -- 172.21.15.28:0/4043215971 learned_addr learned my addr 172.21.15.28:0/4043215971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x560352569a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4327fc640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 -- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 msgr2=0x5603524f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 -- 172.21.15.28:0/4043215971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 msgr2=0x560352569a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x560352569a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.368+0000 7fd4337fe640 1 -- 172.21.15.28:0/4043215971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56035256a0e0 con 0x5603521152f0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd4337fe640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd42c011e50 tx=0x7fd42c02a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e86f39161a6082ea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c003d20 con 0x5603521152f0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 8484+0+0 (secure 0 0 0) 0x7fd42c005540 con 0x5603521152f0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd42c009b30 con 0x5603521152f0 2024-09-21T05:53:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd42c0057c0 con 0x5603521152f0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd44065f740 10 client.26732 fetch_fsmap learned FSMap version 43 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56035256b370 con 0x5603521152f0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd42c009b30 con 0x5603521152f0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 43) ==== 8370+0+0 (secure 0 0 0) 0x7fd42c0274d0 con 0x5603521152f0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.369+0000 7fd44065f740 10 client.26732 fetch_fsmap finished waiting for FSMap version 43 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd44065f740 10 client.26732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd44065f740 20 client.26732 populate_metadata read hostname 'smithi028' 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520d21c0 0x56035256d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd44065f740 4 client.26732 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603524560f0 con 0x5603520d21c0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd4327fc640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520d21c0 0x56035256d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd416ff5640 10 client.26732 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.370+0000 7fd4327fc640 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520d21c0 0x56035256d910 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc6a107cbd31976 server_cookie=4f848071ca08cf1e in_seq=0 out_seq=0 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd416ff5640 1 -- 172.21.15.28:0/4043215971 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603524560f0 con 0x5603520d21c0 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd416ff5640 10 client.26732 handle_command_reply: tid=0 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 client.26732 shutdown 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 msgr2=0x5603524f4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd42c011e50 tx=0x7fd42c02a040 comp rx=0 tx=0).stop 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 shutdown_connections 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5603520d21c0 0x56035256d910 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035213e1b0 0x560352569a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56035211d4d0 0x5603524f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 --2- 172.21.15.28:0/4043215971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5603521152f0 0x5603524f4bc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd416ff5640 0 client.26732 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 >> 172.21.15.28:0/4043215971 conn(0x560352132300 msgr2=0x5603524dfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 shutdown_connections 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.386+0000 7fd44065f740 1 -- 172.21.15.28:0/4043215971 wait complete. 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd44065f740 20 client.26732 trim_cache size 0 max 16384 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-21T05:53:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 msgr2=0x7fd41c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 0x7fd41c0b8b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd4240e49a0 tx=0x7fd4240e9040 comp rx=0 tx=0).stop 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 msgr2=0x7fd43419c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd42400d950 tx=0x7fd42400de10 comp rx=0 tx=0).stop 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 shutdown_connections 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43410dd60 0x7fd4341a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434105250 0x7fd43419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd41c0b6680 0x7fd41c0b8b90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 --2- 172.21.15.28:0/3919435405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd434104900 0x7fd43419c600 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:53:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 >> 172.21.15.28:0/3919435405 conn(0x7fd43407a740 msgr2=0x7fd43410bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:53:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 shutdown_connections 2024-09-21T05:53:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-21T05:53:57.387+0000 7fd4396f2640 1 -- 172.21.15.28:0/3919435405 wait complete. 2024-09-21T05:53:58.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:53:58 smithi129 ceph-mon[34885]: pgmap v533: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-09-21T05:53:58.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:53:58 smithi181 ceph-mon[34819]: pgmap v533: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-09-21T05:53:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:53:58 smithi028 ceph-mon[31227]: pgmap v533: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...0 sec 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB benchmark finished at: Sat Sep 21 05:53:59 2024 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:Results: 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:Benchmark took 335.76 sec 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:Total Results 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout:=============== 2024-09-21T05:53:59.820 INFO:tasks.workunit.client.0.smithi028.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout: ======= ============ ========= ======= =========== ========== 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout: write : 4377600 13038.02 100.000% 100.000% 50.9MB/sec 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout:- 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout:13038.02 Transactions per Second 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout:Throughput Results 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout:=================== 2024-09-21T05:53:59.821 INFO:tasks.workunit.client.0.smithi028.stdout:Write Throughput: 50.9MB/sec 2024-09-21T05:53:59.844 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.844 INFO:tasks.workunit.client.0.smithi028.stdout:System Call Latency statistics in millisecs 2024-09-21T05:53:59.844 INFO:tasks.workunit.client.0.smithi028.stdout:===== 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: Min Avg Max Total Calls 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: ======== ======== ======== ============ 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout:[ open] 0.005000 211.934742 18555.304688 3420 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 19 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 11 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 9 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 22 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 34 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 53 2024-09-21T05:53:59.845 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 205 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 292 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 2 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 11 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 117 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 454 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 1682 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 470 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout:[ write] 0.001000 2.131776 79490.328125 4377600 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 96 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 329 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 460 2024-09-21T05:53:59.846 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 473 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 1372 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 4204 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 16452 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 42248 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 35929 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 43751 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 48119 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 364 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 212 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 226 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 299 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 1336 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 7322 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 53397 2024-09-21T05:53:59.847 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 4120955 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout:[ lseek] 0.000000 0.001141 137.516006 4377600 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 32 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 45 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 28 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 35 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 25 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 21 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 22 2024-09-21T05:53:59.848 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 36 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 121 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 488 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 1776 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 4374969 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout:[ close] 0.001000 0.004345 0.041000 3420 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-21T05:53:59.849 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 4 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 16 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 3400 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout:1.3% User Time 2024-09-21T05:53:59.850 INFO:tasks.workunit.client.0.smithi028.stdout:10.0% System Time 2024-09-21T05:53:59.851 INFO:tasks.workunit.client.0.smithi028.stdout:11.3% CPU Utilization 2024-09-21T05:53:59.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-09-21T05:53:59.853 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -r tmp ffsb 2024-09-21T05:54:00.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:00 smithi181 ceph-mon[34819]: pgmap v534: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 58 KiB/s rd, 8.4 MiB/s wr, 2.08k op/s 2024-09-21T05:54:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:00 smithi028 ceph-mon[31227]: pgmap v534: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 58 KiB/s rd, 8.4 MiB/s wr, 2.08k op/s 2024-09-21T05:54:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:00 smithi129 ceph-mon[34885]: pgmap v534: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 58 KiB/s rd, 8.4 MiB/s wr, 2.08k op/s 2024-09-21T05:54:02.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:02 smithi181 ceph-mon[34819]: pgmap v535: 97 pgs: 97 active+clean; 18 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 29 KiB/s rd, 2.5 MiB/s wr, 638 op/s 2024-09-21T05:54:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:02 smithi028 ceph-mon[31227]: pgmap v535: 97 pgs: 97 active+clean; 18 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 29 KiB/s rd, 2.5 MiB/s wr, 638 op/s 2024-09-21T05:54:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:02 smithi129 ceph-mon[34885]: pgmap v535: 97 pgs: 97 active+clean; 18 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 29 KiB/s rd, 2.5 MiB/s wr, 638 op/s 2024-09-21T05:54:03.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:03 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:03.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:03 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:03.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:03 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:04.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:04 smithi181 ceph-mon[34819]: pgmap v536: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 590 op/s 2024-09-21T05:54:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:04 smithi028 ceph-mon[31227]: pgmap v536: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 590 op/s 2024-09-21T05:54:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:04 smithi129 ceph-mon[34885]: pgmap v536: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 115 KiB/s rd, 2.2 MiB/s wr, 590 op/s 2024-09-21T05:54:05.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:05 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:05.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:05 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:05.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:05 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:06.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:06 smithi181 ceph-mon[34819]: pgmap v537: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 122 KiB/s rd, 1.7 MiB/s wr, 503 op/s 2024-09-21T05:54:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:06 smithi028 ceph-mon[31227]: pgmap v537: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 122 KiB/s rd, 1.7 MiB/s wr, 503 op/s 2024-09-21T05:54:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:06 smithi129 ceph-mon[34885]: pgmap v537: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 122 KiB/s rd, 1.7 MiB/s wr, 503 op/s 2024-09-21T05:54:07.001 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T05:54:07.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T05:54:07.047 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-21T05:54:07.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T05:54:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:07 smithi028 ceph-mon[31227]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T05:54:07.528 DEBUG:teuthology.parallel:result is None 2024-09-21T05:54:07.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T05:54:07.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:07 smithi181 ceph-mon[34819]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T05:54:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:07 smithi129 ceph-mon[34885]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T05:54:08.261 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T05:54:08.261 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-21T05:54:08.271 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:54:08.282 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-21T05:54:08.304 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-21T05:54:08.304 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T05:54:08.304 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:54:08.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-21T05:54:08.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-21T05:54:08.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:08 smithi181 ceph-mon[34819]: pgmap v539: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-09-21T05:54:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:08 smithi028 ceph-mon[31227]: pgmap v539: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-09-21T05:54:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:08 smithi129 ceph-mon[34885]: pgmap v539: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 158 KiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-09-21T05:54:10.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:54:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:10 smithi181 ceph-mon[34819]: pgmap v540: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 351 KiB/s rd, 538 KiB/s wr, 326 op/s 2024-09-21T05:54:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:10 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:10 smithi028 ceph-mon[31227]: pgmap v540: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 351 KiB/s rd, 538 KiB/s wr, 326 op/s 2024-09-21T05:54:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:10 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:10 smithi129 ceph-mon[34885]: pgmap v540: 97 pgs: 97 active+clean; 18 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 351 KiB/s rd, 538 KiB/s wr, 326 op/s 2024-09-21T05:54:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:10 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 -- 172.21.15.28:0/2859175574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814106c20 msgr2=0x7ff814110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 --2- 172.21.15.28:0/2859175574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814106c20 0x7ff814110670 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009ca0 tx=0x7ff80c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 -- 172.21.15.28:0/2859175574 shutdown_connections 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 --2- 172.21.15.28:0/2859175574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814106c20 0x7ff814110670 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 --2- 172.21.15.28:0/2859175574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff8141066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 --2- 172.21.15.28:0/2859175574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814101ff0 0x7ff814102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 -- 172.21.15.28:0/2859175574 >> 172.21.15.28:0/2859175574 conn(0x7ff814077cc0 msgr2=0x7ff814078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 -- 172.21.15.28:0/2859175574 shutdown_connections 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.214+0000 7ff81b179640 1 -- 172.21.15.28:0/2859175574 wait complete. 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.215+0000 7ff81b179640 1 Processor -- start 2024-09-21T05:54:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.215+0000 7ff81b179640 1 -- start start 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814101ff0 0x7ff81419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff81419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 0x7ff81419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff814113ae0 con 0x7ff814101ff0 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff814113960 con 0x7ff8141062d0 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff81b179640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff814113c60 con 0x7ff814106c20 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff819716640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 0x7ff81419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff818f15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814101ff0 0x7ff81419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff819716640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 0x7ff81419fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46054/0 (socket says 172.21.15.28:46054) 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff819716640 1 -- 172.21.15.28:0/629703248 learned_addr learned my addr 172.21.15.28:0/629703248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff81419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 -- 172.21.15.28:0/629703248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 msgr2=0x7ff81419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 0x7ff81419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 -- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814101ff0 msgr2=0x7ff81419a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814101ff0 0x7ff81419a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.216+0000 7ff813fff640 1 -- 172.21.15.28:0/629703248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8141a03c0 con 0x7ff8141062d0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff813fff640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff81419a9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff814075ad0 tx=0x7ff80000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff811ffb640 1 -- 172.21.15.28:0/629703248 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80000cc40 con 0x7ff8141062d0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8141a06b0 con 0x7ff8141062d0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff811ffb640 1 -- 172.21.15.28:0/629703248 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff80000cde0 con 0x7ff8141062d0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff811ffb640 1 -- 172.21.15.28:0/629703248 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800018800 con 0x7ff8141062d0 2024-09-21T05:54:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.217+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a6e10 con 0x7ff8141062d0 2024-09-21T05:54:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.220+0000 7ff811ffb640 1 -- 172.21.15.28:0/629703248 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff8000189a0 con 0x7ff8141062d0 2024-09-21T05:54:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.220+0000 7ff8216ed740 1 Processor -- start 2024-09-21T05:54:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 -- start start 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cf51b0 0x55de530c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de5303be50 con 0x55de52d1e070 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55de5302b280 con 0x55de52cf51b0 2024-09-21T05:54:11.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff8216ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55de5302f090 con 0x55de52cfd390 2024-09-21T05:54:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff819716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff819716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42866/0 (socket says 172.21.15.28:42866) 2024-09-21T05:54:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff819716640 1 -- 172.21.15.28:0/3664254829 learned_addr learned my addr 172.21.15.28:0/3664254829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff818f15640 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cf51b0 0x55de530c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.221+0000 7ff813fff640 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 -- 172.21.15.28:0/3664254829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 msgr2=0x55de530c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 -- 172.21.15.28:0/3664254829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cf51b0 msgr2=0x55de530c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cf51b0 0x55de530c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 -- 172.21.15.28:0/3664254829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de52d89650 con 0x55de52d1e070 2024-09-21T05:54:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff811ffb640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 0x7ff8040b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff819716640 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff80c02f450 tx=0x7ff80c02f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9bc7659c16a79 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff818f15640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 0x7ff8040b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff811ffb640 1 -- 172.21.15.28:0/629703248 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 8572+0+0 (secure 0 0 0) 0x7ff8000e7a40 con 0x7ff8141062d0 2024-09-21T05:54:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff7feffd640 1 -- 172.21.15.28:0/3664254829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c031cf0 con 0x55de52d1e070 2024-09-21T05:54:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff7feffd640 1 -- 172.21.15.28:0/3664254829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff80c036070 con 0x55de52d1e070 2024-09-21T05:54:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.222+0000 7ff7feffd640 1 -- 172.21.15.28:0/3664254829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0042a0 con 0x55de52d1e070 2024-09-21T05:54:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 -- 172.21.15.28:0/3664254829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 msgr2=0x55de530c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff80c02f450 tx=0x7ff80c02f480 comp rx=0 tx=0).stop 2024-09-21T05:54:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff818f15640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 0x7ff8040b8c90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55de530c42d0 tx=0x7ff808005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 -- 172.21.15.28:0/3664254829 shutdown_connections 2024-09-21T05:54:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cf51b0 0x55de530c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 --2- 172.21.15.28:0/3664254829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530c5650 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.223+0000 7ff8216ed740 1 -- 172.21.15.28:0/3664254829 >> 172.21.15.28:0/3664254829 conn(0x55de52d12460 msgr2=0x55de530c3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.224+0000 7ff8216ed740 1 -- 172.21.15.28:0/3664254829 shutdown_connections 2024-09-21T05:54:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.224+0000 7ff8216ed740 1 -- 172.21.15.28:0/3664254829 wait complete. 2024-09-21T05:54:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.224+0000 7ff8216ed740 1 -- start start 2024-09-21T05:54:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.224+0000 7ff8216ed740 1 Processor -- start 2024-09-21T05:54:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530d1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530fef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de52e8c700 con 0x55de52cf51b0 2024-09-21T05:54:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55de52fc6110 con 0x55de52d1e070 2024-09-21T05:54:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55de530d3c50 con 0x55de52cfd390 2024-09-21T05:54:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff8216ed740 10 client.0 _ll_register_callbacks cb 0x55de52fdf990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff818f15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff818f15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42876/0 (socket says 172.21.15.28:42876) 2024-09-21T05:54:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff818f15640 1 -- 172.21.15.28:0/772219753 learned_addr learned my addr 172.21.15.28:0/772219753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff813fff640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.225+0000 7ff819716640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530fef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 -- 172.21.15.28:0/772219753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 msgr2=0x55de530fef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530fef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 -- 172.21.15.28:0/772219753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 msgr2=0x55de530d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 -- 172.21.15.28:0/772219753 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de52ff3c40 con 0x55de52cfd390 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff818f15640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff813fff640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530d1310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff8000e5100 tx=0x7ff8000acee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef6597d03023d23 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff7f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8000ad200 con 0x55de52cfd390 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff80c036070 con 0x55de52cfd390 2024-09-21T05:54:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(74..74 src has 1..74) ==== 8572+0+0 (secure 0 0 0) 0x7ff8000ac780 con 0x55de52cfd390 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff800010b00 con 0x55de52cfd390 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff8216ed740 10 client.26740 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.226+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55de530e4f40 con 0x55de52cfd390 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff80c036070 con 0x55de52cfd390 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff8000b6760 con 0x55de52cfd390 2024-09-21T05:54:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 10 client.26740 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 10 client.26740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 20 client.26740 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cdc9f0 0x55de530e8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 4 client.26740 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de5303dda0 con 0x55de52cdc9f0 2024-09-21T05:54:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.227+0000 7ff818f15640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cdc9f0 0x55de530e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.228+0000 7ff7f3fff640 10 client.26740 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.228+0000 7ff818f15640 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cdc9f0 0x55de530e8360 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0a72704b94beae7 server_cookie=3ca04d7a0dfcd666 in_seq=0 out_seq=0 2024-09-21T05:54:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.228+0000 7ff7f3fff640 1 -- 172.21.15.28:0/772219753 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55de5303dda0 con 0x55de52cdc9f0 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.228+0000 7ff7f3fff640 10 client.26740 handle_command_reply: tid=0 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.228+0000 7ff8216ed740 1 client.26740 shutdown 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 msgr2=0x55de530d1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530d1310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff8000e5100 tx=0x7ff8000acee0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 shutdown_connections 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cdc9f0 0x55de530e8360 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cfd390 0x55de530d1310 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff7f3fff640 0 client.26740 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 --2- 172.21.15.28:0/772219753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cf51b0 0x55de530d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 >> 172.21.15.28:0/772219753 conn(0x55de52d12460 msgr2=0x55de530c8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 shutdown_connections 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 1 -- 172.21.15.28:0/772219753 wait complete. 2024-09-21T05:54:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.229+0000 7ff8216ed740 20 client.26740 trim_cache size 0 max 16384 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.249+0000 7ff8216ed740 1 Processor -- start 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.249+0000 7ff8216ed740 1 -- start start 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cfd390 0x55de530ff100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff814073370 con 0x55de52d1e070 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff814074fd0 con 0x55de52cfd390 2024-09-21T05:54:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff8216ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff814073aa0 con 0x55de52cf51b0 2024-09-21T05:54:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff819716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff818f15640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff818f15640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:46086/0 (socket says 172.21.15.28:46086) 2024-09-21T05:54:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff818f15640 1 -- 172.21.15.28:0/962211820 learned_addr learned my addr 172.21.15.28:0/962211820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff813fff640 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cfd390 0x55de530ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.250+0000 7ff818f15640 1 -- 172.21.15.28:0/962211820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cfd390 msgr2=0x55de530ff100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff818f15640 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cfd390 0x55de530ff100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff818f15640 1 -- 172.21.15.28:0/962211820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 msgr2=0x55de530cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff818f15640 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff818f15640 1 -- 172.21.15.28:0/962211820 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de530c7940 con 0x55de52cf51b0 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff818f15640 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55de530e8aa0 tx=0x7ff80801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2580993d25972c66 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff7fe7fc640 1 -- 172.21.15.28:0/962211820 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808004490 con 0x55de52cf51b0 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff7fe7fc640 1 -- 172.21.15.28:0/962211820 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8080054d0 con 0x55de52cf51b0 2024-09-21T05:54:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.251+0000 7ff7fe7fc640 1 -- 172.21.15.28:0/962211820 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808003850 con 0x55de52cf51b0 2024-09-21T05:54:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- 172.21.15.28:0/962211820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 msgr2=0x55de530c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55de530e8aa0 tx=0x7ff80801b000 comp rx=0 tx=0).stop 2024-09-21T05:54:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- 172.21.15.28:0/962211820 shutdown_connections 2024-09-21T05:54:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52d1e070 0x55de530cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52cfd390 0x55de530ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 --2- 172.21.15.28:0/962211820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530c3700 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- 172.21.15.28:0/962211820 >> 172.21.15.28:0/962211820 conn(0x55de52d12460 msgr2=0x55de530fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- 172.21.15.28:0/962211820 shutdown_connections 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- 172.21.15.28:0/962211820 wait complete. 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.252+0000 7ff8216ed740 1 -- start start 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 Processor -- start 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8140748d0 con 0x55de52cfd390 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8140741a0 con 0x55de52d1e070 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff814071370 con 0x55de52cf51b0 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff813fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff813fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42886/0 (socket says 172.21.15.28:42886) 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff813fff640 1 -- 172.21.15.28:0/2480830832 learned_addr learned my addr 172.21.15.28:0/2480830832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff819716640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff8216ed740 10 client.0 _ll_register_callbacks cb 0x7ff8000a9d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.253+0000 7ff818f15640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 -- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 msgr2=0x55de530f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530f6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 -- 172.21.15.28:0/2480830832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 msgr2=0x55de530f7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530f7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 -- 172.21.15.28:0/2480830832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de5315dbc0 con 0x55de52cfd390 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff819716640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff813fff640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff8000e7dc0 tx=0x7ff8000e5170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f6ac1775608a0e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff7f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:11.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8000ad020 con 0x55de52cfd390 2024-09-21T05:54:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(74..74 src has 1..74) ==== 8572+0+0 (secure 0 0 0) 0x7ff8000b3470 con 0x55de52cfd390 2024-09-21T05:54:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8080054d0 con 0x55de52cfd390 2024-09-21T05:54:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff8000ad200 con 0x55de52cfd390 2024-09-21T05:54:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff8216ed740 10 client.17160 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.254+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55de5315ee30 con 0x55de52cfd390 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8080054d0 con 0x55de52cfd390 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff8000b3670 con 0x55de52cfd390 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 10 client.17160 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 10 client.17160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 20 client.17160 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cb1da0 0x55de53161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 4 client.17160 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de53039850 con 0x55de52cb1da0 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.255+0000 7ff818f15640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cb1da0 0x55de53161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff7f3fff640 10 client.17160 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff818f15640 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cb1da0 0x55de53161650 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a0a8110074106f server_cookie=f90860d1c5fb4449 in_seq=0 out_seq=0 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff7f3fff640 1 -- 172.21.15.28:0/2480830832 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2930 (crc 0 0 0) 0x55de53039850 con 0x55de52cb1da0 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff7f3fff640 10 client.17160 handle_command_reply: tid=0 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 client.17160 shutdown 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 msgr2=0x55de530f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff8000e7dc0 tx=0x7ff8000e5170 comp rx=0 tx=0).stop 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 shutdown_connections 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55de52cb1da0 0x55de53161650 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55de52d1e070 0x55de530f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de52cfd390 0x55de530f6fc0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 --2- 172.21.15.28:0/2480830832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55de52cf51b0 0x55de530f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 >> 172.21.15.28:0/2480830832 conn(0x55de52d12460 msgr2=0x55de530d2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.256+0000 7ff7f3fff640 0 client.17160 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.257+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 shutdown_connections 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.257+0000 7ff8216ed740 1 -- 172.21.15.28:0/2480830832 wait complete. 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.257+0000 7ff8216ed740 20 client.17160 trim_cache size 0 max 16384 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:54:11.264 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.265 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:54:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.268 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.269 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:11.270 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 msgr2=0x7ff8040b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 0x7ff8040b8c90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55de530c42d0 tx=0x7ff808005f90 comp rx=0 tx=0).stop 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 msgr2=0x7ff81419a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff81419a9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff814075ad0 tx=0x7ff80000ee50 comp rx=0 tx=0).stop 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 shutdown_connections 2024-09-21T05:54:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff814106c20 0x7ff81419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8141062d0 0x7ff81419a9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff8040b6780 0x7ff8040b8c90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 --2- 172.21.15.28:0/629703248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814101ff0 0x7ff81419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 >> 172.21.15.28:0/629703248 conn(0x7ff814077cc0 msgr2=0x7ff814107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 shutdown_connections 2024-09-21T05:54:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:11.258+0000 7ff81b179640 1 -- 172.21.15.28:0/629703248 wait complete. 2024-09-21T05:54:12.141 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:54:12.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:54:12.407 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:12.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:12 smithi028 ceph-mon[31227]: pgmap v541: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 338 KiB/s rd, 438 KiB/s wr, 329 op/s 2024-09-21T05:54:12.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:12 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:12.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:12 smithi181 ceph-mon[34819]: pgmap v541: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 338 KiB/s rd, 438 KiB/s wr, 329 op/s 2024-09-21T05:54:12.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:12 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:12 smithi129 ceph-mon[34885]: pgmap v541: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 338 KiB/s rd, 438 KiB/s wr, 329 op/s 2024-09-21T05:54:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:12 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 -- 172.21.15.28:0/374518188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4108cf0 msgr2=0x7f1ed410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 --2- 172.21.15.28:0/374518188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4108cf0 0x7f1ed410f520 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009ca0 tx=0x7f1ecc02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 -- 172.21.15.28:0/374518188 shutdown_connections 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 --2- 172.21.15.28:0/374518188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4108cf0 0x7f1ed410f520 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 --2- 172.21.15.28:0/374518188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 --2- 172.21.15.28:0/374518188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4102a60 0x7f1ed4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 -- 172.21.15.28:0/374518188 >> 172.21.15.28:0/374518188 conn(0x7f1ed4079730 msgr2=0x7f1ed4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 -- 172.21.15.28:0/374518188 shutdown_connections 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.943+0000 7f1edc69e640 1 -- 172.21.15.28:0/374518188 wait complete. 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 Processor -- start 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 -- start start 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 0x7f1ed419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4113a70 con 0x7f1ed4102a60 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41138f0 con 0x7f1ed4108cf0 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.944+0000 7f1edc69e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4113bf0 con 0x7f1ed41083a0 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42900/0 (socket says 172.21.15.28:42900) 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 -- 172.21.15.28:0/1193113589 learned_addr learned my addr 172.21.15.28:0/1193113589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1ed9c39640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1edac3b640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 0x7f1ed419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 -- 172.21.15.28:0/1193113589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 msgr2=0x7f1ed419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 -- 172.21.15.28:0/1193113589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 msgr2=0x7f1ed419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 0x7f1ed419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 -- 172.21.15.28:0/1193113589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed41a0370 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1ed9c39640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1edac3b640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 0x7f1ed419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1eda43a640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e9f0 tx=0x7f1ed000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed000cd80 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ed00108e0 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0018980 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed41a0660 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.945+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed41a6dc0 con 0x7f1ed4102a60 2024-09-21T05:54:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.946+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed4102e70 con 0x7f1ed4102a60 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.948+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1ed0010430 con 0x7f1ed4102a60 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.949+0000 7f1ecb7fe640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 0x7f1ec00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.949+0000 7f1ed9c39640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 0x7f1ec00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.949+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f1ed00e7ff0 con 0x7f1ed4102a60 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.950+0000 7f1ed9c39640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 0x7f1ec00b8b90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ed406a2b0 tx=0x7f1ec4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:12.952+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1ed00ae500 con 0x7f1ed4102a60 2024-09-21T05:54:13.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.188+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ed410ab20 con 0x7f1ed4102a60 2024-09-21T05:54:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.189+0000 7f1ecb7fe640 1 -- 172.21.15.28:0/1193113589 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11995 (secure 0 0 0) 0x7f1ed00b28e0 con 0x7f1ed4102a60 2024-09-21T05:54:13.192 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:54:13.192 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"btime":"2024-09-21T05:54:11:183838+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:54:11.055265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:54:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-09-21T05:54:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 msgr2=0x7f1ec00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 0x7f1ec00b8b90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ed406a2b0 tx=0x7f1ec4005f90 comp rx=0 tx=0).stop 2024-09-21T05:54:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 msgr2=0x7f1ed419a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e9f0 tx=0x7f1ed000eeb0 comp rx=0 tx=0).stop 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 shutdown_connections 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ed4108cf0 0x7f1ed419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ed41083a0 0x7f1ed419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ec00b6680 0x7f1ec00b8b90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 --2- 172.21.15.28:0/1193113589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ed4102a60 0x7f1ed419a400 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 >> 172.21.15.28:0/1193113589 conn(0x7f1ed4079730 msgr2=0x7f1ed41043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 shutdown_connections 2024-09-21T05:54:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:13.192+0000 7f1edc69e640 1 -- 172.21.15.28:0/1193113589 wait complete. 2024-09-21T05:54:13.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:13 smithi181 ceph-mon[34819]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T05:54:13.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:13 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1193113589' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:13 smithi028 ceph-mon[31227]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T05:54:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:13 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1193113589' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:13 smithi129 ceph-mon[34885]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T05:54:13.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:13 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1193113589' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:13.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:54:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:14.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:14 smithi028 ceph-mon[31227]: pgmap v543: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 290 KiB/s rd, 252 KiB/s wr, 290 op/s 2024-09-21T05:54:14.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:14 smithi181 ceph-mon[34819]: pgmap v543: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 290 KiB/s rd, 252 KiB/s wr, 290 op/s 2024-09-21T05:54:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:14 smithi129 ceph-mon[34885]: pgmap v543: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 290 KiB/s rd, 252 KiB/s wr, 290 op/s 2024-09-21T05:54:14.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/2484017891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c4109df0 msgr2=0x7fd3c4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/2484017891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c4109df0 0x7fd3c4110620 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009ca0 tx=0x7fd3bc02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/2484017891 shutdown_connections 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/2484017891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c4109df0 0x7fd3c4110620 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/2484017891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c41094a0 0x7fd3c41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/2484017891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 0x7fd3c4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/2484017891 >> 172.21.15.28:0/2484017891 conn(0x7fd3c407b850 msgr2=0x7fd3c407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/2484017891 shutdown_connections 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/2484017891 wait complete. 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.701+0000 7fd3c9c0d640 1 Processor -- start 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 -- start start 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 0x7fd3c419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c4109df0 0x7fd3c41a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4113ab0 con 0x7fd3c41094a0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4113930 con 0x7fd3c4103b60 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c9c0d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4113c30 con 0x7fd3c4109df0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42910/0 (socket says 172.21.15.28:42910) 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3436974490 learned_addr learned my addr 172.21.15.28:0/3436974490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 0x7fd3c419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c4109df0 0x7fd3c41a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3436974490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c4109df0 msgr2=0x7fd3c41a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c4109df0 0x7fd3c41a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3436974490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 msgr2=0x7fd3c419c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.702+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 0x7fd3c419c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3436974490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c41a25f0 con 0x7fd3c41094a0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800e9c0 tx=0x7fd3b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c0ff9640 1 -- 172.21.15.28:0/3436974490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b800ccc0 con 0x7fd3c41094a0 2024-09-21T05:54:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c0ff9640 1 -- 172.21.15.28:0/3436974490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b80054d0 con 0x7fd3c41094a0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c41a28e0 con 0x7fd3c41094a0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c0ff9640 1 -- 172.21.15.28:0/3436974490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8010430 con 0x7fd3c41094a0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.703+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c41a9040 con 0x7fd3c41094a0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.705+0000 7fd3d0181740 1 Processor -- start 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 -- start start 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 0x55ce5f813be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f814120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f789ea0 con 0x55ce5f46c070 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f7792d0 con 0x55ce5f4431b0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3d0181740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f77d0e0 con 0x55ce5f44b390 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c0ff9640 1 -- 172.21.15.28:0/3436974490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd3b80105d0 con 0x7fd3c41094a0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42912/0 (socket says 172.21.15.28:42912) 2024-09-21T05:54:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 -- 172.21.15.28:0/3519117858 learned_addr learned my addr 172.21.15.28:0/3519117858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f814120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 0x55ce5f813be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 -- 172.21.15.28:0/3519117858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 msgr2=0x55ce5f814120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f814120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 -- 172.21.15.28:0/3519117858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 msgr2=0x55ce5f813be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 0x55ce5f813be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c3fff640 1 -- 172.21.15.28:0/3519117858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce5f4d7650 con 0x55ce5f46c070 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.706+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 0x55ce5f813be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc005cc0 tx=0x7fd3bc02f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7455503f48bb706e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3a9ffb640 1 -- 172.21.15.28:0/3519117858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc031cf0 con 0x55ce5f46c070 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3a9ffb640 1 -- 172.21.15.28:0/3519117858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3bc036040 con 0x55ce5f46c070 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3a9ffb640 1 -- 172.21.15.28:0/3519117858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc004040 con 0x55ce5f46c070 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3c0ff9640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 0x7fd3ac0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 0x7fd3ac0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.707+0000 7fd3c0ff9640 1 -- 172.21.15.28:0/3436974490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fd3b80e7ac0 con 0x7fd3c41094a0 2024-09-21T05:54:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- 172.21.15.28:0/3519117858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 msgr2=0x55ce5f8136a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 0x7fd3ac0b8c10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55ce5f812320 tx=0x7fd3b400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc005cc0 tx=0x7fd3bc02f420 comp rx=0 tx=0).stop 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- 172.21.15.28:0/3519117858 shutdown_connections 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f814120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f4431b0 0x55ce5f813be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3519117858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f8136a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- 172.21.15.28:0/3519117858 >> 172.21.15.28:0/3519117858 conn(0x55ce5f460460 msgr2=0x55ce5f811590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- 172.21.15.28:0/3519117858 shutdown_connections 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- 172.21.15.28:0/3519117858 wait complete. 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.708+0000 7fd3d0181740 1 -- start start 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 Processor -- start 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f81f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f46c070 0x55ce5f82b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f5da700 con 0x55ce5f4431b0 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f714160 con 0x55ce5f44b390 2024-09-21T05:54:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ce5f821e20 con 0x55ce5f46c070 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3d0181740 10 client.0 _ll_register_callbacks cb 0x55ce5f72d9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42928/0 (socket says 172.21.15.28:42928) 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 -- 172.21.15.28:0/1293440904 learned_addr learned my addr 172.21.15.28:0/1293440904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f81f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f46c070 0x55ce5f82b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 -- 172.21.15.28:0/1293440904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f46c070 msgr2=0x55ce5f82b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f46c070 0x55ce5f82b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 -- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 msgr2=0x55ce5f81f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f81f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.709+0000 7fd3c37fe640 1 -- 172.21.15.28:0/1293440904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce5f741c90 con 0x55ce5f4431b0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40126d0 tx=0x7fd3b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48748635925d7ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400f9a0 con 0x55ce5f4431b0 2024-09-21T05:54:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fd3b400fb40 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3bc036040 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd3b400fd40 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3d0181740 10 client.17178 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ce5f82daf0 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3bc036040 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.710+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fd3b401f4b0 con 0x55ce5f4431b0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 10 client.17178 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 10 client.17178 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 20 client.17178 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f42a9f0 0x55ce5f830fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 4 client.17178 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce5f78bdf0 con 0x55ce5f42a9f0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f42a9f0 0x55ce5f830fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f42a9f0 0x55ce5f830fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9aed1b241da12d1 server_cookie=a8e625019779315f in_seq=0 out_seq=0 2024-09-21T05:54:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.711+0000 7fd3a2ffd640 10 client.17178 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/1293440904 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ce5f78bdf0 con 0x55ce5f42a9f0 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3a2ffd640 10 client.17178 handle_command_reply: tid=0 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 client.17178 shutdown 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 msgr2=0x55ce5f81ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40126d0 tx=0x7fd3b40046c0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 shutdown_connections 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f42a9f0 0x55ce5f830fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f46c070 0x55ce5f82b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f81f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3a2ffd640 0 client.17178 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1293440904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f81ed80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 >> 172.21.15.28:0/1293440904 conn(0x55ce5f460460 msgr2=0x55ce5f816850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 shutdown_connections 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 1 -- 172.21.15.28:0/1293440904 wait complete. 2024-09-21T05:54:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.712+0000 7fd3d0181740 20 client.17178 trim_cache size 0 max 16384 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 Processor -- start 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 -- start start 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f4431b0 0x55ce5f8209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f810bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b1a10 con 0x55ce5f46c070 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b3670 con 0x55ce5f44b390 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3d0181740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b2140 con 0x55ce5f4431b0 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42942/0 (socket says 172.21.15.28:42942) 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 -- 172.21.15.28:0/1065032493 learned_addr learned my addr 172.21.15.28:0/1065032493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f4431b0 0x55ce5f8209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f810bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 -- 172.21.15.28:0/1065032493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f4431b0 msgr2=0x55ce5f8209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f4431b0 0x55ce5f8209b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 -- 172.21.15.28:0/1065032493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 msgr2=0x55ce5f810bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f810bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c3fff640 1 -- 172.21.15.28:0/1065032493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce5f815bc0 con 0x55ce5f46c070 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.726+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f810bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.727+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc02f500 tx=0x7fd3bc009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bc761718eb42f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.727+0000 7fd3a97fa640 1 -- 172.21.15.28:0/1065032493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc04a040 con 0x55ce5f46c070 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.727+0000 7fd3a97fa640 1 -- 172.21.15.28:0/1065032493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3bc053070 con 0x55ce5f46c070 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.727+0000 7fd3a97fa640 1 -- 172.21.15.28:0/1065032493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc004260 con 0x55ce5f46c070 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- 172.21.15.28:0/1065032493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 msgr2=0x55ce5f811100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc02f500 tx=0x7fd3bc009430 comp rx=0 tx=0).stop 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- 172.21.15.28:0/1065032493 shutdown_connections 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f46c070 0x55ce5f811100 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f44b390 0x55ce5f810bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 --2- 172.21.15.28:0/1065032493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f4431b0 0x55ce5f8209b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- 172.21.15.28:0/1065032493 >> 172.21.15.28:0/1065032493 conn(0x55ce5f460460 msgr2=0x55ce5f8202a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- 172.21.15.28:0/1065032493 shutdown_connections 2024-09-21T05:54:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- 172.21.15.28:0/1065032493 wait complete. 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 -- start start 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.728+0000 7fd3d0181740 1 Processor -- start 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f822f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f46c070 0x55ce5f823640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b2f70 con 0x55ce5f4431b0 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41b2840 con 0x55ce5f46c070 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c41af9f0 con 0x55ce5f44b390 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3d0181740 10 client.0 _ll_register_callbacks cb 0x7fd3b4005cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42944/0 (socket says 172.21.15.28:42944) 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c37fe640 1 -- 172.21.15.28:0/3146903754 learned_addr learned my addr 172.21.15.28:0/3146903754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f822f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c3fff640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f46c070 0x55ce5f823640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f46c070 msgr2=0x55ce5f823640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f46c070 0x55ce5f823640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3146903754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 msgr2=0x55ce5f827a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 -- 172.21.15.28:0/3146903754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce5f8a31b0 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3c2ffd640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f822f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80e51d0 tx=0x7fd3b80e9f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499cb7c07955f140 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.729+0000 7fd3a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80ad040 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fd3b80acb30 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3bc053070 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd3b800ce60 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 10 client.26770 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ce5f8a4250 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3bc053070 con 0x55ce5f44b390 2024-09-21T05:54:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fd3b80108c0 con 0x55ce5f44b390 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 10 client.26770 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 10 client.26770 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 20 client.26770 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f3ffda0 0x55ce5f8a6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 4 client.26770 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce5f7878a0 con 0x55ce5f3ffda0 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.730+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f3ffda0 0x55ce5f8a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3c37fe640 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f3ffda0 0x55ce5f8a6990 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9809c326f1b1c5c5 server_cookie=12828c8aa7a7d42c in_seq=0 out_seq=0 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3a2ffd640 10 client.26770 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3a2ffd640 1 -- 172.21.15.28:0/3146903754 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce5f7878a0 con 0x55ce5f3ffda0 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3a2ffd640 10 client.26770 handle_command_reply: tid=0 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 client.26770 shutdown 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 msgr2=0x55ce5f822f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f822f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80e51d0 tx=0x7fd3b80e9f10 comp rx=0 tx=0).stop 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 shutdown_connections 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55ce5f3ffda0 0x55ce5f8a6990 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce5f46c070 0x55ce5f823640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce5f44b390 0x55ce5f822f70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3a2ffd640 0 client.26770 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 --2- 172.21.15.28:0/3146903754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce5f4431b0 0x55ce5f827a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 >> 172.21.15.28:0/3146903754 conn(0x55ce5f460460 msgr2=0x55ce5f820040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 shutdown_connections 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 1 -- 172.21.15.28:0/3146903754 wait complete. 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.731+0000 7fd3d0181740 20 client.26770 trim_cache size 0 max 16384 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 msgr2=0x7fd3ac0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 0x7fd3ac0b8c10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55ce5f812320 tx=0x7fd3b400b390 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 msgr2=0x7fd3c419cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800e9c0 tx=0x7fd3b800ee80 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 shutdown_connections 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3c4109df0 0x7fd3c41a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd3ac0b6700 0x7fd3ac0b8c10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c41094a0 0x7fd3c419cc20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 --2- 172.21.15.28:0/3436974490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c4103b60 0x7fd3c419c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 >> 172.21.15.28:0/3436974490 conn(0x7fd3c407b850 msgr2=0x7fd3c41072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 shutdown_connections 2024-09-21T05:54:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:14.732+0000 7fd3c9c0d640 1 -- 172.21.15.28:0/3436974490 wait complete. 2024-09-21T05:54:15.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:54:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:16.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:16 smithi028 ceph-mon[31227]: pgmap v544: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 255 KiB/s rd, 228 KiB/s wr, 277 op/s 2024-09-21T05:54:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 -- 172.21.15.28:0/2831303476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 msgr2=0x7fb7d4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/2831303476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d4104dd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009900 tx=0x7fb7cc02a410 comp rx=0 tx=0).stop 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 -- 172.21.15.28:0/2831303476 shutdown_connections 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/2831303476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d410de20 0x7fb7d4110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/2831303476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d4105310 0x7fb7d410d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/2831303476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d4104dd0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 -- 172.21.15.28:0/2831303476 >> 172.21.15.28:0/2831303476 conn(0x7fb7d4079730 msgr2=0x7fb7d4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 -- 172.21.15.28:0/2831303476 shutdown_connections 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.460+0000 7fb7dbb74640 1 -- 172.21.15.28:0/2831303476 wait complete. 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 Processor -- start 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 -- start start 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d4105310 0x7fb7d419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 0x7fb7d419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d4113a80 con 0x7fb7d41049c0 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d4113900 con 0x7fb7d4105310 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7dbb74640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d4113c00 con 0x7fb7d410de20 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7d9910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7d9910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56786/0 (socket says 172.21.15.28:56786) 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7d9910640 1 -- 172.21.15.28:0/1987840308 learned_addr learned my addr 172.21.15.28:0/1987840308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.461+0000 7fb7d910f640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d4105310 0x7fb7d419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7da111640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 0x7fb7d419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 -- 172.21.15.28:0/1987840308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 msgr2=0x7fb7d419fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 0x7fb7d419fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 -- 172.21.15.28:0/1987840308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d4105310 msgr2=0x7fb7d419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d4105310 0x7fb7d419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 -- 172.21.15.28:0/1987840308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d41a04a0 con 0x7fb7d41049c0 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7da111640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 0x7fb7d419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc031a90 tx=0x7fb7cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d2ffd640 1 -- 172.21.15.28:0/1987840308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc03f030 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d41a0730 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d2ffd640 1 -- 172.21.15.28:0/1987840308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7cc003cb0 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7d2ffd640 1 -- 172.21.15.28:0/1987840308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc035070 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.462+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d41a6ef0 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 Processor -- start 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 -- start start 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7d2ffd640 1 -- 172.21.15.28:0/1987840308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb7cc042030 con 0x7fb7d41049c0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef595a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef595abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef5592390 0x558ef595b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ef58d0ea0 con 0x558ef558a1b0 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ef58c02d0 con 0x558ef55b3070 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558ef58c40e0 con 0x558ef5592390 2024-09-21T05:54:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.465+0000 7fb7d9910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef595abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7da111640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef595a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7da111640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef595a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:48748/0 (socket says 172.21.15.28:48748) 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7da111640 1 -- 172.21.15.28:0/1936903345 learned_addr learned my addr 172.21.15.28:0/1936903345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d910f640 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef5592390 0x558ef595b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 -- 172.21.15.28:0/1936903345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef5592390 msgr2=0x558ef595b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef5592390 0x558ef595b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 -- 172.21.15.28:0/1936903345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 msgr2=0x558ef595a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef595a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 -- 172.21.15.28:0/1936903345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef561e650 con 0x558ef558a1b0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d2ffd640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 0x7fb7bc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef595abe0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc105470 tx=0x7fb7cc10bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c868cc490ed7c629 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7da111640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 0x7fb7bc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.466+0000 7fb7d2ffd640 1 -- 172.21.15.28:0/1987840308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb7cc107d20 con 0x7fb7d41049c0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.467+0000 7fb7bbfff640 1 -- 172.21.15.28:0/1936903345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0ca1c0 con 0x558ef558a1b0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.467+0000 7fb7bbfff640 1 -- 172.21.15.28:0/1936903345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7cc0cd680 con 0x558ef558a1b0 2024-09-21T05:54:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.467+0000 7fb7bbfff640 1 -- 172.21.15.28:0/1936903345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0cd2e0 con 0x558ef558a1b0 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1936903345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 msgr2=0x558ef595abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef595abe0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc105470 tx=0x7fb7cc10bf90 comp rx=0 tx=0).stop 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7da111640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 0x7fb7bc0b8b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4008670 tx=0x7fb7c4011040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1936903345 shutdown_connections 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef5592390 0x558ef595b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef595abe0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1936903345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef595a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1936903345 >> 172.21.15.28:0/1936903345 conn(0x558ef55a7460 msgr2=0x558ef5958590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1936903345 shutdown_connections 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1936903345 wait complete. 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 -- start start 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.468+0000 7fb7e20e8740 1 Processor -- start 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef5592390 0x558ef59662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ef5721700 con 0x558ef558a1b0 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ef585b160 con 0x558ef5592390 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558ef5968c40 con 0x558ef55b3070 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7e20e8740 10 client.0 _ll_register_callbacks cb 0x558ef58749e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56806/0 (socket says 172.21.15.28:56806) 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 -- 172.21.15.28:0/1118128267 learned_addr learned my addr 172.21.15.28:0/1118128267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d910f640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef5592390 0x558ef59662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7da111640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 -- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 msgr2=0x558ef5971ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 -- 172.21.15.28:0/1118128267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef5592390 msgr2=0x558ef59662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef5592390 0x558ef59662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7d9910640 1 -- 172.21.15.28:0/1118128267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef5888c90 con 0x558ef558a1b0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.469+0000 7fb7da111640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7d9910640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc105440 tx=0x7fb7cc03b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8fb10ff327c1f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc049030 con 0x558ef558a1b0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb7cc03bc50 con 0x558ef558a1b0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7cc0cd680 con 0x558ef558a1b0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb7cc0cd4c0 con 0x558ef558a1b0 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7e20e8740 10 client.17196 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558ef59748c0 con 0x558ef558a1b0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7cc0cd680 con 0x558ef558a1b0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.470+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb7cc1056f0 con 0x558ef558a1b0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 10 client.17196 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 10 client.17196 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 20 client.17196 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef55719f0 0x558ef5977c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 4 client.17196 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ef58d2df0 con 0x558ef55719f0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.471+0000 7fb7d910f640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef55719f0 0x558ef5977c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.472+0000 7fb7b97fa640 10 client.17196 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.472+0000 7fb7d910f640 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef55719f0 0x558ef5977c20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8baba4a034ddd10f server_cookie=e81d4d6d171ae8bc in_seq=0 out_seq=0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.472+0000 7fb7b97fa640 1 -- 172.21.15.28:0/1118128267 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558ef58d2df0 con 0x558ef55719f0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.472+0000 7fb7b97fa640 10 client.17196 handle_command_reply: tid=0 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.472+0000 7fb7e20e8740 1 client.17196 shutdown 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 msgr2=0x558ef5965db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc105440 tx=0x7fb7cc03b690 comp rx=0 tx=0).stop 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 shutdown_connections 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef55719f0 0x558ef5977c20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef5592390 0x558ef59662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/1118128267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef558a1b0 0x558ef5965db0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7b97fa640 0 client.17196 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 >> 172.21.15.28:0/1118128267 conn(0x558ef55a7460 msgr2=0x558ef595d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 shutdown_connections 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 1 -- 172.21.15.28:0/1118128267 wait complete. 2024-09-21T05:54:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.473+0000 7fb7e20e8740 20 client.17196 trim_cache size 0 max 16384 2024-09-21T05:54:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.486+0000 7fb7e20e8740 1 Processor -- start 2024-09-21T05:54:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.486+0000 7fb7e20e8740 1 -- start start 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef558a1b0 0x558ef595c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41ad5f0 con 0x558ef5592390 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41af250 con 0x558ef558a1b0 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41add20 con 0x558ef55b3070 2024-09-21T05:54:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56816/0 (socket says 172.21.15.28:56816) 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 -- 172.21.15.28:0/327860381 learned_addr learned my addr 172.21.15.28:0/327860381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d9910640 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef558a1b0 0x558ef595c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7da111640 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 -- 172.21.15.28:0/327860381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 msgr2=0x558ef5971c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 -- 172.21.15.28:0/327860381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef558a1b0 msgr2=0x558ef595c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef558a1b0 0x558ef595c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 -- 172.21.15.28:0/327860381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef595d500 con 0x558ef5592390 2024-09-21T05:54:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.487+0000 7fb7d910f640 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80058c0 tx=0x7fb7c800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b5e1812efae86e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7bb7fe640 1 -- 172.21.15.28:0/327860381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8005220 con 0x558ef5592390 2024-09-21T05:54:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7bb7fe640 1 -- 172.21.15.28:0/327860381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8004c50 con 0x558ef5592390 2024-09-21T05:54:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7bb7fe640 1 -- 172.21.15.28:0/327860381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80258d0 con 0x558ef5592390 2024-09-21T05:54:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 -- 172.21.15.28:0/327860381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 msgr2=0x558ef59679e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80058c0 tx=0x7fb7c800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 -- 172.21.15.28:0/327860381 shutdown_connections 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef55b3070 0x558ef5971c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef59679e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/327860381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef558a1b0 0x558ef595c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 -- 172.21.15.28:0/327860381 >> 172.21.15.28:0/327860381 conn(0x558ef55a7460 msgr2=0x558ef59672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 -- 172.21.15.28:0/327860381 shutdown_connections 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.488+0000 7fb7e20e8740 1 -- 172.21.15.28:0/327860381 wait complete. 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 -- start start 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 Processor -- start 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef558a1b0 0x558ef596c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef59e1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41aeb50 con 0x558ef5592390 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41ae420 con 0x558ef55b3070 2024-09-21T05:54:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41ab5d0 con 0x558ef558a1b0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.489+0000 7fb7e20e8740 10 client.0 _ll_register_callbacks cb 0x7fb7c400f830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56832/0 (socket says 172.21.15.28:56832) 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 -- 172.21.15.28:0/43042991 learned_addr learned my addr 172.21.15.28:0/43042991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7da111640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef59e1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d9910640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef558a1b0 0x558ef596c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 -- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef558a1b0 msgr2=0x558ef596c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef558a1b0 0x558ef596c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 -- 172.21.15.28:0/43042991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 msgr2=0x558ef59e1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef59e1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 -- 172.21.15.28:0/43042991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef5960a50 con 0x558ef5592390 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7d910f640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800ad60 tx=0x7fb7c8004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2d5cb63191b1fa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035020 con 0x558ef5592390 2024-09-21T05:54:16.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb7c8005230 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.490+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7c8004c50 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb7c8040050 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 10 client.17202 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558ef59e2a20 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8004c50 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb7c8031470 con 0x558ef5592390 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 10 client.17202 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 10 client.17202 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 20 client.17202 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef5546da0 0x558ef59e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 4 client.17202 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.491+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ef58ce8a0 con 0x558ef5546da0 2024-09-21T05:54:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7d9910640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef5546da0 0x558ef59e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7d9910640 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef5546da0 0x558ef59e52c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=148eab3a72dcc526 server_cookie=ed1c2173ef43cce2 in_seq=0 out_seq=0 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7b97fa640 10 client.17202 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7b97fa640 1 -- 172.21.15.28:0/43042991 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ef58ce8a0 con 0x558ef5546da0 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7b97fa640 10 client.17202 handle_command_reply: tid=0 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 client.17202 shutdown 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 msgr2=0x558ef596c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800ad60 tx=0x7fb7c8004960 comp rx=0 tx=0).stop 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 shutdown_connections 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x558ef5546da0 0x558ef59e52c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ef55b3070 0x558ef59e1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ef5592390 0x558ef596c950 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 --2- 172.21.15.28:0/43042991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558ef558a1b0 0x558ef596c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7b97fa640 0 client.17202 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.492+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 >> 172.21.15.28:0/43042991 conn(0x558ef55a7460 msgr2=0x558ef59679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 shutdown_connections 2024-09-21T05:54:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7e20e8740 1 -- 172.21.15.28:0/43042991 wait complete. 2024-09-21T05:54:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7e20e8740 20 client.17202 trim_cache size 0 max 16384 2024-09-21T05:54:16.499 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:54:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 msgr2=0x7fb7bc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 0x7fb7bc0b8b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4008670 tx=0x7fb7c4011040 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 msgr2=0x7fb7d419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc031a90 tx=0x7fb7cc004270 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 shutdown_connections 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d410de20 0x7fb7d419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7d4105310 0x7fb7d419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb7bc0b6680 0x7fb7bc0b8b90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 --2- 172.21.15.28:0/1987840308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d41049c0 0x7fb7d419a530 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 >> 172.21.15.28:0/1987840308 conn(0x7fb7d4079730 msgr2=0x7fb7d4100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 shutdown_connections 2024-09-21T05:54:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:16.493+0000 7fb7dbb74640 1 -- 172.21.15.28:0/1987840308 wait complete. 2024-09-21T05:54:16.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:16 smithi181 ceph-mon[34819]: pgmap v544: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 255 KiB/s rd, 228 KiB/s wr, 277 op/s 2024-09-21T05:54:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:16 smithi129 ceph-mon[34885]: pgmap v544: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 255 KiB/s rd, 228 KiB/s wr, 277 op/s 2024-09-21T05:54:17.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:54:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 -- 172.21.15.28:0/511476384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b01094a0 msgr2=0x7fb2b01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/511476384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b01094a0 0x7fb2b01098b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009900 tx=0x7fb2a402a410 comp rx=0 tx=0).stop 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 -- 172.21.15.28:0/511476384 shutdown_connections 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/511476384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0109df0 0x7fb2b0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/511476384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b01094a0 0x7fb2b01098b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/511476384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b0103b60 0x7fb2b0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 -- 172.21.15.28:0/511476384 >> 172.21.15.28:0/511476384 conn(0x7fb2b007b850 msgr2=0x7fb2b007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 -- 172.21.15.28:0/511476384 shutdown_connections 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.926+0000 7fb2b61ca640 1 -- 172.21.15.28:0/511476384 wait complete. 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 Processor -- start 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 -- start start 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0103b60 0x7fb2b019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b01094a0 0x7fb2b019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b0113ad0 con 0x7fb2b0109df0 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b0113950 con 0x7fb2b0103b60 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2b61ca640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b0113c50 con 0x7fb2b01094a0 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56844/0 (socket says 172.21.15.28:56844) 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2affff640 1 -- 172.21.15.28:0/3566228546 learned_addr learned my addr 172.21.15.28:0/3566228546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.927+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b01094a0 0x7fb2b019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0103b60 0x7fb2b019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 -- 172.21.15.28:0/3566228546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b01094a0 msgr2=0x7fb2b019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b01094a0 0x7fb2b019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 -- 172.21.15.28:0/3566228546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0103b60 msgr2=0x7fb2b019a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0103b60 0x7fb2b019a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 -- 172.21.15.28:0/3566228546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b01a04d0 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2affff640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800eaa0 tx=0x7fb2a800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2acff9640 1 -- 172.21.15.28:0/3566228546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8004d20 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b01a07c0 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2acff9640 1 -- 172.21.15.28:0/3566228546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a8005020 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2acff9640 1 -- 172.21.15.28:0/3566228546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a80116b0 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.928+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b01a6f20 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.931+0000 7fb2bc73e740 1 Processor -- start 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.931+0000 7fb2acff9640 1 -- 172.21.15.28:0/3566228546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb2a80051d0 con 0x7fb2b0109df0 2024-09-21T05:54:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.931+0000 7fb2bc73e740 1 -- start start 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a7391b0 0x55a53ab09be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:48798/0 (socket says 172.21.15.28:48798) 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab0a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a53aa7fea0 con 0x55a53a741390 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2aeffd640 1 -- 172.21.15.28:0/972504499 learned_addr learned my addr 172.21.15.28:0/972504499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a53aa6f2d0 con 0x55a53a762070 2024-09-21T05:54:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.932+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a53aa730e0 con 0x55a53a7391b0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab0a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a7391b0 0x55a53ab09be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 -- 172.21.15.28:0/972504499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a7391b0 msgr2=0x55a53ab09be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a7391b0 0x55a53ab09be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 -- 172.21.15.28:0/972504499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 msgr2=0x55a53ab0a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab0a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 -- 172.21.15.28:0/972504499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53a7cd650 con 0x55a53a762070 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb2b006c9f0 tx=0x7fb2a4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1eb8a9ed607e098 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2acff9640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 0x7fb2980b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb295ffb640 1 -- 172.21.15.28:0/972504499 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4044020 con 0x55a53a762070 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 0x7fb2980b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb295ffb640 1 -- 172.21.15.28:0/972504499 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a4031820 con 0x55a53a762070 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.933+0000 7fb2acff9640 1 -- 172.21.15.28:0/3566228546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb2a80e78e0 con 0x7fb2b0109df0 2024-09-21T05:54:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb295ffb640 1 -- 172.21.15.28:0/972504499 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4035040 con 0x55a53a762070 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 msgr2=0x55a53ab096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 0x7fb2980b8b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x55a53ab08320 tx=0x7fb2a0008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb2b006c9f0 tx=0x7fb2a4000bc0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 shutdown_connections 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab0a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a7391b0 0x55a53ab09be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/972504499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a762070 0x55a53ab096a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.934+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 >> 172.21.15.28:0/972504499 conn(0x55a53a756460 msgr2=0x55a53ab07590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.935+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 shutdown_connections 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.935+0000 7fb2bc73e740 1 -- 172.21.15.28:0/972504499 wait complete. 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.935+0000 7fb2bc73e740 1 -- start start 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.935+0000 7fb2bc73e740 1 Processor -- start 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab190c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a53a8d0700 con 0x55a53a741390 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a53aa0a160 con 0x55a53a7391b0 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a53ab1c170 con 0x55a53a762070 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2bc73e740 10 client.0 _ll_register_callbacks cb 0x55a53aa239e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56870/0 (socket says 172.21.15.28:56870) 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 -- 172.21.15.28:0/1555937499 learned_addr learned my addr 172.21.15.28:0/1555937499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2affff640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab19b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 -- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 msgr2=0x55a53ab19b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab19b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 -- 172.21.15.28:0/1555937499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 msgr2=0x55a53ab190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab190c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 -- 172.21.15.28:0/1555937499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53aa37c90 con 0x55a53a741390 2024-09-21T05:54:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2affff640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4032d40 tx=0x7fb2a40319d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd55e20ead83d4bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.936+0000 7fb28effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4044030 con 0x55a53a741390 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a4031820 con 0x55a53a741390 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb2a4004450 con 0x55a53a741390 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb2a4035040 con 0x55a53a741390 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 10 client.17214 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a53ab27e10 con 0x55a53a741390 2024-09-21T05:54:17.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a4031820 con 0x55a53a741390 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb2a405a610 con 0x55a53a741390 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 10 client.17214 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 10 client.17214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.937+0000 7fb2bc73e740 20 client.17214 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.938+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a7209f0 0x55a53ab2b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.938+0000 7fb2bc73e740 4 client.17214 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.938+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a53aa81df0 con 0x55a53a7209f0 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.938+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a7209f0 0x55a53ab2b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.938+0000 7fb28effd640 10 client.17214 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.939+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a7209f0 0x55a53ab2b170 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5befdf1b5282f55a server_cookie=a251529108c4c49a in_seq=0 out_seq=0 2024-09-21T05:54:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.939+0000 7fb28effd640 1 -- 172.21.15.28:0/1555937499 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a53aa81df0 con 0x55a53a7209f0 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.939+0000 7fb28effd640 10 client.17214 handle_command_reply: tid=0 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.939+0000 7fb2bc73e740 1 client.17214 shutdown 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 msgr2=0x55a53ab19600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4032d40 tx=0x7fb2a40319d0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 shutdown_connections 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a7209f0 0x55a53ab2b170 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab19600 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/1555937499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 >> 172.21.15.28:0/1555937499 conn(0x55a53a756460 msgr2=0x55a53ab0b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb28effd640 0 client.17214 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 shutdown_connections 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 1 -- 172.21.15.28:0/1555937499 wait complete. 2024-09-21T05:54:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.940+0000 7fb2bc73e740 20 client.17214 trim_cache size 0 max 16384 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.953+0000 7fb2bc73e740 1 Processor -- start 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.953+0000 7fb2bc73e740 1 -- start start 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.953+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab1b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.953+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01ad520 con 0x55a53a741390 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01af180 con 0x55a53a7391b0 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01adc50 con 0x55a53a762070 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56886/0 (socket says 172.21.15.28:56886) 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 -- 172.21.15.28:0/3728071452 learned_addr learned my addr 172.21.15.28:0/3728071452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:17.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab1b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2affff640 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 -- 172.21.15.28:0/3728071452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 msgr2=0x55a53ab167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 -- 172.21.15.28:0/3728071452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 msgr2=0x55a53ab1b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab1b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 -- 172.21.15.28:0/3728071452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53a720c20 con 0x55a53a741390 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb2a402a450 tx=0x7fb2a40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db0782c228244f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2957fa640 1 -- 172.21.15.28:0/3728071452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4044040 con 0x55a53a741390 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2957fa640 1 -- 172.21.15.28:0/3728071452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a40315a0 con 0x55a53a741390 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.954+0000 7fb2957fa640 1 -- 172.21.15.28:0/3728071452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a40035f0 con 0x55a53a741390 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 -- 172.21.15.28:0/3728071452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 msgr2=0x55a53ab162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb2a402a450 tx=0x7fb2a40058d0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 -- 172.21.15.28:0/3728071452 shutdown_connections 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a762070 0x55a53ab167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a741390 0x55a53ab162b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/3728071452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab1b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 -- 172.21.15.28:0/3728071452 >> 172.21.15.28:0/3728071452 conn(0x55a53a756460 msgr2=0x55a53ab0b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 -- 172.21.15.28:0/3728071452 shutdown_connections 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.955+0000 7fb2bc73e740 1 -- 172.21.15.28:0/3728071452 wait complete. 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 -- start start 2024-09-21T05:54:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 Processor -- start 2024-09-21T05:54:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab2c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a741390 0x55a53ab2c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01aea80 con 0x55a53a762070 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01ae350 con 0x55a53a7391b0 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b01ab500 con 0x55a53a741390 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.956+0000 7fb2bc73e740 10 client.0 _ll_register_callbacks cb 0x7fb2a4032f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56900/0 (socket says 172.21.15.28:56900) 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 -- 172.21.15.28:0/688237414 learned_addr learned my addr 172.21.15.28:0/688237414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2aeffd640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a741390 0x55a53ab2c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab2c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 -- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a741390 msgr2=0x55a53ab2c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a741390 0x55a53ab2c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 -- 172.21.15.28:0/688237414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 msgr2=0x55a53ab2c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab2c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 -- 172.21.15.28:0/688237414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53ab948f0 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab2c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2affff640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80e5090 tx=0x7fb2a80acf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b9800c5c26cbbf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb28effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a801f070 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb2a80a9080 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.957+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a40315a0 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb2a80b2210 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 10 client.17220 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a53ab95b30 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a40315a0 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb2a800f040 con 0x55a53a762070 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 10 client.17220 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 10 client.17220 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:54:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 20 client.17220 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a6f5da0 0x55a53ab98300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 4 client.17220 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a53aa7d8a0 con 0x55a53a6f5da0 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.958+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a6f5da0 0x55a53ab98300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb28effd640 10 client.17220 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2af7fe640 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a6f5da0 0x55a53ab98300 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a037d338613b67b8 server_cookie=dd9405ab4bb3d9d9 in_seq=0 out_seq=0 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb28effd640 1 -- 172.21.15.28:0/688237414 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a53aa7d8a0 con 0x55a53a6f5da0 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb28effd640 10 client.17220 handle_command_reply: tid=0 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 client.17220 shutdown 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 msgr2=0x55a53ab2cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80e5090 tx=0x7fb2a80acf60 comp rx=0 tx=0).stop 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 shutdown_connections 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55a53a6f5da0 0x55a53ab98300 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53a762070 0x55a53ab2cb80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a53a741390 0x55a53ab2c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 --2- 172.21.15.28:0/688237414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a53a7391b0 0x55a53ab2c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 >> 172.21.15.28:0/688237414 conn(0x55a53a756460 msgr2=0x55a53ab0b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb28effd640 0 client.17220 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 shutdown_connections 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.959+0000 7fb2bc73e740 1 -- 172.21.15.28:0/688237414 wait complete. 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2bc73e740 20 client.17220 trim_cache size 0 max 16384 2024-09-21T05:54:17.966 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 msgr2=0x7fb2980b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 0x7fb2980b8b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x55a53ab08320 tx=0x7fb2a0008040 comp rx=0 tx=0).stop 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 msgr2=0x7fb2b019fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800eaa0 tx=0x7fb2a800eea0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 shutdown_connections 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb2980b6680 0x7fb2980b8b90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b0109df0 0x7fb2b019fda0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2b01094a0 0x7fb2b019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 --2- 172.21.15.28:0/3566228546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2b0103b60 0x7fb2b019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 >> 172.21.15.28:0/3566228546 conn(0x7fb2b007b850 msgr2=0x7fb2b01075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.960+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 shutdown_connections 2024-09-21T05:54:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:17.961+0000 7fb2b61ca640 1 -- 172.21.15.28:0/3566228546 wait complete. 2024-09-21T05:54:18.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:18 smithi181 ceph-mon[34819]: pgmap v545: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 222 KiB/s rd, 199 KiB/s wr, 248 op/s 2024-09-21T05:54:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:18 smithi028 ceph-mon[31227]: pgmap v545: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 222 KiB/s rd, 199 KiB/s wr, 248 op/s 2024-09-21T05:54:18.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:54:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:18 smithi129 ceph-mon[34885]: pgmap v545: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 222 KiB/s rd, 199 KiB/s wr, 248 op/s 2024-09-21T05:54:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- 172.21.15.28:0/2064668283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38102290 msgr2=0x7fbe381026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/2064668283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38102290 0x7fbe381026a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009900 tx=0x7fbe3402a410 comp rx=0 tx=0).stop 2024-09-21T05:54:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- 172.21.15.28:0/2064668283 shutdown_connections 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/2064668283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38108520 0x7fbe3810ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/2064668283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 0x7fbe38107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/2064668283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38102290 0x7fbe381026a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- 172.21.15.28:0/2064668283 >> 172.21.15.28:0/2064668283 conn(0x7fbe38079ea0 msgr2=0x7fbe3807a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- 172.21.15.28:0/2064668283 shutdown_connections 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- 172.21.15.28:0/2064668283 wait complete. 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 Processor -- start 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.319+0000 7fbe3fedf640 1 -- start start 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38102290 0x7fbe3819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 0x7fbe3819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381121d0 con 0x7fbe38108520 2024-09-21T05:54:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112050 con 0x7fbe38107bd0 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3fedf640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112350 con 0x7fbe38102290 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56924/0 (socket says 172.21.15.28:56924) 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3dc7b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38102290 0x7fbe3819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 -- 172.21.15.28:0/1339184838 learned_addr learned my addr 172.21.15.28:0/1339184838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 0x7fbe3819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 -- 172.21.15.28:0/1339184838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38102290 msgr2=0x7fbe3819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38102290 0x7fbe3819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 -- 172.21.15.28:0/1339184838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 msgr2=0x7fbe3819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 0x7fbe3819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.320+0000 7fbe3e47c640 1 -- 172.21.15.28:0/1339184838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe381a2770 con 0x7fbe38108520 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe3e47c640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000cad0 tx=0x7fbe3000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe2effd640 1 -- 172.21.15.28:0/1339184838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000ae20 con 0x7fbe38108520 2024-09-21T05:54:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe2effd640 1 -- 172.21.15.28:0/1339184838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe30005020 con 0x7fbe38108520 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe381a2a60 con 0x7fbe38108520 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe2effd640 1 -- 172.21.15.28:0/1339184838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000f6b0 con 0x7fbe38108520 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.321+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe381a91c0 con 0x7fbe38108520 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.324+0000 7fbe46453740 1 Processor -- start 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.324+0000 7fbe2effd640 1 -- 172.21.15.28:0/1339184838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbe3000f850 con 0x7fbe38108520 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.324+0000 7fbe46453740 1 -- start start 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f4981be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4982120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583f48f7ea0 con 0x5583f45da070 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5583f48e72d0 con 0x5583f45b11b0 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe46453740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5583f48eb0e0 con 0x5583f45b9390 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56930/0 (socket says 172.21.15.28:56930) 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 -- 172.21.15.28:0/484643754 learned_addr learned my addr 172.21.15.28:0/484643754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f4981be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4982120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe2effd640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 0x7fbe240b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 -- 172.21.15.28:0/484643754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 msgr2=0x5583f4982120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4982120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 -- 172.21.15.28:0/484643754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 msgr2=0x5583f4981be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f4981be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3d47a640 1 -- 172.21.15.28:0/484643754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583f4645650 con 0x5583f45da070 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.325+0000 7fbe3e47c640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 0x7fbe240b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.326+0000 7fbe2effd640 1 -- 172.21.15.28:0/1339184838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fbe300e75d0 con 0x7fbe38108520 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.326+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ae30 tx=0x7fbe2800a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c9fee021d4067d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.326+0000 7fbe23fff640 1 -- 172.21.15.28:0/484643754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004020 con 0x5583f45da070 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.326+0000 7fbe23fff640 1 -- 172.21.15.28:0/484643754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28017030 con 0x5583f45da070 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.326+0000 7fbe23fff640 1 -- 172.21.15.28:0/484643754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28012450 con 0x5583f45da070 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- 172.21.15.28:0/484643754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 msgr2=0x5583f49816a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe3e47c640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 0x7fbe240b8c10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe300cf380 tx=0x7fbe300ddec0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ae30 tx=0x7fbe2800a6e0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- 172.21.15.28:0/484643754 shutdown_connections 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4982120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f4981be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 --2- 172.21.15.28:0/484643754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f49816a0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- 172.21.15.28:0/484643754 >> 172.21.15.28:0/484643754 conn(0x5583f45ce460 msgr2=0x5583f497f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- 172.21.15.28:0/484643754 shutdown_connections 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- 172.21.15.28:0/484643754 wait complete. 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.327+0000 7fbe46453740 1 -- start start 2024-09-21T05:54:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 Processor -- start 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f498d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45da070 0x5583f498d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583f4748700 con 0x5583f45b11b0 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5583f4882160 con 0x5583f45b9390 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5583f498fe80 con 0x5583f45da070 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe46453740 10 client.0 _ll_register_callbacks cb 0x5583f489b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56938/0 (socket says 172.21.15.28:56938) 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 -- 172.21.15.28:0/2818584987 learned_addr learned my addr 172.21.15.28:0/2818584987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f498d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 -- 172.21.15.28:0/2818584987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45da070 msgr2=0x5583f498d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45da070 0x5583f498d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.328+0000 7fbe3d47a640 1 -- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 msgr2=0x5583f498d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f498d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe3d47a640 1 -- 172.21.15.28:0/2818584987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583f48afc90 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f498d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005e80 tx=0x7fbe2800a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dafd972311ae31d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280037f0 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fbe28003e80 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe28017030 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbe28004080 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe46453740 10 client.17232 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5583f499bb50 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.329+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28017030 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fbe2801cc80 con 0x5583f45b11b0 2024-09-21T05:54:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 10 client.17232 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 10 client.17232 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 20 client.17232 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f45989f0 0x5583f499ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 4 client.17232 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583f48f9df0 con 0x5583f45989f0 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f45989f0 0x5583f499ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f45989f0 0x5583f499ef10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a61b50b67e356fd4 server_cookie=421e1365009f949d in_seq=0 out_seq=0 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.330+0000 7fbe217fa640 10 client.17232 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.331+0000 7fbe217fa640 1 -- 172.21.15.28:0/2818584987 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5583f48f9df0 con 0x5583f45989f0 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.331+0000 7fbe217fa640 10 client.17232 handle_command_reply: tid=0 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.331+0000 7fbe46453740 1 client.17232 shutdown 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.331+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 msgr2=0x5583f498cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.331+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005e80 tx=0x7fbe2800a980 comp rx=0 tx=0).stop 2024-09-21T05:54:19.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 shutdown_connections 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f45989f0 0x5583f499ef10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45da070 0x5583f498d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe217fa640 0 client.17232 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f498d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 --2- 172.21.15.28:0/2818584987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45b11b0 0x5583f498cd40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 >> 172.21.15.28:0/2818584987 conn(0x5583f45ce460 msgr2=0x5583f4984880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 shutdown_connections 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 1 -- 172.21.15.28:0/2818584987 wait complete. 2024-09-21T05:54:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.332+0000 7fbe46453740 20 client.17232 trim_cache size 0 max 16384 2024-09-21T05:54:19.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 Processor -- start 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 -- start start 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b11b0 0x5583f498d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f497ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381b1ae0 con 0x5583f45da070 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381b3740 con 0x5583f45b9390 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.345+0000 7fbe46453740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381b2210 con 0x5583f45b11b0 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56954/0 (socket says 172.21.15.28:56954) 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 -- 172.21.15.28:0/4229259828 learned_addr learned my addr 172.21.15.28:0/4229259828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f497ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b11b0 0x5583f498d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 -- 172.21.15.28:0/4229259828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b11b0 msgr2=0x5583f498d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b11b0 0x5583f498d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 -- 172.21.15.28:0/4229259828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 msgr2=0x5583f497ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f497ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 -- 172.21.15.28:0/4229259828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583f498f320 con 0x5583f45da070 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005b10 tx=0x7fbe2800ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e740b9b6f7c2133e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe237fe640 1 -- 172.21.15.28:0/4229259828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28035070 con 0x5583f45da070 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.346+0000 7fbe237fe640 1 -- 172.21.15.28:0/4229259828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28017040 con 0x5583f45da070 2024-09-21T05:54:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe237fe640 1 -- 172.21.15.28:0/4229259828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28029cb0 con 0x5583f45da070 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 -- 172.21.15.28:0/4229259828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 msgr2=0x5583f497f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005b10 tx=0x7fbe2800ab90 comp rx=0 tx=0).stop 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 -- 172.21.15.28:0/4229259828 shutdown_connections 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f497f100 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b9390 0x5583f497ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 --2- 172.21.15.28:0/4229259828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b11b0 0x5583f498d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 -- 172.21.15.28:0/4229259828 >> 172.21.15.28:0/4229259828 conn(0x5583f45ce460 msgr2=0x5583f498caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 -- 172.21.15.28:0/4229259828 shutdown_connections 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.347+0000 7fbe46453740 1 -- 172.21.15.28:0/4229259828 wait complete. 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 -- start start 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 Processor -- start 2024-09-21T05:54:19.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f49910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4991600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381b3040 con 0x5583f45da070 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381b2910 con 0x5583f45b11b0 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381afac0 con 0x5583f45b9390 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe46453740 10 client.0 _ll_register_callbacks cb 0x7fbe2800ac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.348+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56958/0 (socket says 172.21.15.28:56958) 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 -- 172.21.15.28:0/1488664780 learned_addr learned my addr 172.21.15.28:0/1488664780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f49910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4991600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 -- 172.21.15.28:0/1488664780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 msgr2=0x5583f4991600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4991600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 -- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 msgr2=0x5583f49910c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f49910c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 -- 172.21.15.28:0/1488664780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583f4a086e0 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe3d47a640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ae30 tx=0x7fbe280291c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c92b2e694105f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28035040 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fbe28003e20 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe28017040 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbe28029360 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe46453740 10 client.17238 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.349+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5583f4a09940 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28017040 con 0x5583f45da070 2024-09-21T05:54:19.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fbe28048510 con 0x5583f45da070 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 10 client.17238 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 10 client.17238 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 20 client.17238 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f456dda0 0x5583f4a0c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 4 client.17238 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5583f48f58a0 con 0x5583f456dda0 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.350+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f456dda0 0x5583f4a0c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe3dc7b640 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f456dda0 0x5583f4a0c030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1388bb9e9d88ad01 server_cookie=c91523e3f17d6aa8 in_seq=0 out_seq=0 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe217fa640 10 client.17238 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe217fa640 1 -- 172.21.15.28:0/1488664780 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5583f48f58a0 con 0x5583f456dda0 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe217fa640 10 client.17238 handle_command_reply: tid=0 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 client.17238 shutdown 2024-09-21T05:54:19.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 msgr2=0x5583f4991b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ae30 tx=0x7fbe280291c0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 shutdown_connections 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x5583f456dda0 0x5583f4a0c030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583f45da070 0x5583f4991b40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5583f45b9390 0x5583f4991600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe217fa640 0 client.17238 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 --2- 172.21.15.28:0/1488664780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5583f45b11b0 0x5583f49910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 >> 172.21.15.28:0/1488664780 conn(0x5583f45ce460 msgr2=0x5583f4982680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 shutdown_connections 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 1 -- 172.21.15.28:0/1488664780 wait complete. 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.351+0000 7fbe46453740 20 client.17238 trim_cache size 0 max 16384 2024-09-21T05:54:19.358 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:54:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 msgr2=0x7fbe240b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 0x7fbe240b8c10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe300cf380 tx=0x7fbe300ddec0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 msgr2=0x7fbe381a2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000cad0 tx=0x7fbe3000cf90 comp rx=0 tx=0).stop 2024-09-21T05:54:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 shutdown_connections 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbe240b6700 0x7fbe240b8c10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe38108520 0x7fbe381a2040 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe38107bd0 0x7fbe3819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 --2- 172.21.15.28:0/1339184838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe38102290 0x7fbe3819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 >> 172.21.15.28:0/1339184838 conn(0x7fbe38079ea0 msgr2=0x7fbe380fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 shutdown_connections 2024-09-21T05:54:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:19.352+0000 7fbe3fedf640 1 -- 172.21.15.28:0/1339184838 wait complete. 2024-09-21T05:54:20.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:54:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:20.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:20 smithi181 ceph-mon[34819]: pgmap v546: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 125 op/s 2024-09-21T05:54:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:20 smithi028 ceph-mon[31227]: pgmap v546: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 125 op/s 2024-09-21T05:54:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:20 smithi129 ceph-mon[34885]: pgmap v546: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 125 op/s 2024-09-21T05:54:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 -- 172.21.15.28:0/3553435625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494109df0 msgr2=0x7f1494110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 --2- 172.21.15.28:0/3553435625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494109df0 0x7f1494110620 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f148c009ca0 tx=0x7f148c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 -- 172.21.15.28:0/3553435625 shutdown_connections 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 --2- 172.21.15.28:0/3553435625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494109df0 0x7f1494110620 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 --2- 172.21.15.28:0/3553435625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 0x7f14941098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 --2- 172.21.15.28:0/3553435625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494103b60 0x7f1494103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 -- 172.21.15.28:0/3553435625 >> 172.21.15.28:0/3553435625 conn(0x7f149407b850 msgr2=0x7f149407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 -- 172.21.15.28:0/3553435625 shutdown_connections 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.985+0000 7f1499b28640 1 -- 172.21.15.28:0/3553435625 wait complete. 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 Processor -- start 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 -- start start 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 0x7f149419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 0x7f149419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1494113ab0 con 0x7f1494103b60 2024-09-21T05:54:20.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1494113930 con 0x7f14941094a0 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f1499b28640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1494113c30 con 0x7f1494109df0 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f14937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f14937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56972/0 (socket says 172.21.15.28:56972) 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.986+0000 7f14937fe640 1 -- 172.21.15.28:0/2133212406 learned_addr learned my addr 172.21.15.28:0/2133212406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 0x7f149419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1493fff640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 0x7f149419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 -- 172.21.15.28:0/2133212406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 msgr2=0x7f149419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 0x7f149419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 -- 172.21.15.28:0/2133212406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 msgr2=0x7f149419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:20.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 0x7f149419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 -- 172.21.15.28:0/2133212406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14941a0390 con 0x7f1494103b60 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1493fff640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 0x7f149419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f14937fe640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f148400c970 tx=0x7f148400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1490ff9640 1 -- 172.21.15.28:0/2133212406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484007bb0 con 0x7f1494103b60 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1490ff9640 1 -- 172.21.15.28:0/2133212406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1484007d50 con 0x7f1494103b60 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14941a0680 con 0x7f1494103b60 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1490ff9640 1 -- 172.21.15.28:0/2133212406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148400f450 con 0x7f1494103b60 2024-09-21T05:54:20.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.987+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14941a6de0 con 0x7f1494103b60 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.990+0000 7f14a009c740 1 Processor -- start 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 -- start start 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa356a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa35be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f1490ff9640 1 -- 172.21.15.28:0/2133212406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1484004f20 con 0x7f1494103b60 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55636a9abea0 con 0x55636a66d390 2024-09-21T05:54:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636a99b2d0 con 0x55636a68e070 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14a009c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55636a99f0e0 con 0x55636a6651b0 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f14937fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa35be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56984/0 (socket says 172.21.15.28:56984) 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f1492ffd640 1 -- 172.21.15.28:0/2296636010 learned_addr learned my addr 172.21.15.28:0/2296636010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.991+0000 7f1493fff640 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 -- 172.21.15.28:0/2296636010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 msgr2=0x55636aa35be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa35be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 -- 172.21.15.28:0/2296636010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 msgr2=0x55636aa356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa356a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 -- 172.21.15.28:0/2296636010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636a6f9650 con 0x55636a66d390 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f148800bc10 tx=0x7f148800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111f4364f2867d69 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1479ffb640 1 -- 172.21.15.28:0/2296636010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488015070 con 0x55636a66d390 2024-09-21T05:54:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1490ff9640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 0x7f147c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1479ffb640 1 -- 172.21.15.28:0/2296636010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148801a910 con 0x55636a66d390 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.992+0000 7f1479ffb640 1 -- 172.21.15.28:0/2296636010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880117d0 con 0x55636a66d390 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.993+0000 7f1493fff640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 0x7f147c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.993+0000 7f1490ff9640 1 -- 172.21.15.28:0/2133212406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f14840e74e0 con 0x7f1494103b60 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.993+0000 7f14a009c740 1 -- 172.21.15.28:0/2296636010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 msgr2=0x55636aa36120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.993+0000 7f1493fff640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 0x7f147c0b8b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f148c004af0 tx=0x7f148c005d80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.993+0000 7f14a009c740 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f148800bc10 tx=0x7f148800f730 comp rx=0 tx=0).stop 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 -- 172.21.15.28:0/2296636010 shutdown_connections 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa36120 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa35be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 --2- 172.21.15.28:0/2296636010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa356a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 -- 172.21.15.28:0/2296636010 >> 172.21.15.28:0/2296636010 conn(0x55636a682460 msgr2=0x55636aa33590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 -- 172.21.15.28:0/2296636010 shutdown_connections 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 -- 172.21.15.28:0/2296636010 wait complete. 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 -- start start 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.994+0000 7f14a009c740 1 Processor -- start 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa40d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa51580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55636a7fc700 con 0x55636a66d390 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636a936160 con 0x55636a6651b0 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55636aa48290 con 0x55636a68e070 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14a009c740 10 client.0 _ll_register_callbacks cb 0x55636a94f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57000/0 (socket says 172.21.15.28:57000) 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 -- 172.21.15.28:0/2279198863 learned_addr learned my addr 172.21.15.28:0/2279198863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14937fe640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa40d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1493fff640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa51580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 -- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 msgr2=0x55636aa51580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa51580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 -- 172.21.15.28:0/2279198863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 msgr2=0x55636aa40d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa40d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f1492ffd640 1 -- 172.21.15.28:0/2279198863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636a963c90 con 0x55636a66d390 2024-09-21T05:54:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.995+0000 7f14937fe640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa40d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1488006010 tx=0x7f148800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f1d7be3152613b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488015070 con 0x55636a66d390 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f148801aac0 con 0x55636a66d390 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f148801a910 con 0x55636a66d390 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f148801acc0 con 0x55636a66d390 2024-09-21T05:54:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 10 client.17250 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55636aa53fa0 con 0x55636a66d390 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148801a910 con 0x55636a66d390 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f1488005480 con 0x55636a66d390 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 10 client.17250 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 20 client.17250 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a64c9f0 0x55636aa573b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 4 client.17250 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.996+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55636a9addf0 con 0x55636a64c9f0 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.997+0000 7f14937fe640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a64c9f0 0x55636aa573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.997+0000 7f1472ffd640 10 client.17250 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.997+0000 7f14937fe640 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a64c9f0 0x55636aa573b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28ff17f262feb0f3 server_cookie=bf63038cfd72ff43 in_seq=0 out_seq=0 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f1472ffd640 1 -- 172.21.15.28:0/2279198863 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55636a9addf0 con 0x55636a64c9f0 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f1472ffd640 10 client.17250 handle_command_reply: tid=0 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 client.17250 shutdown 2024-09-21T05:54:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 msgr2=0x55636aa41290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1488006010 tx=0x7f148800bfa0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 shutdown_connections 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a64c9f0 0x55636aa573b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa51580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa41290 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 --2- 172.21.15.28:0/2279198863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa40d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f1472ffd640 0 client.17250 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 >> 172.21.15.28:0/2279198863 conn(0x55636a682460 msgr2=0x55636aa38880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 shutdown_connections 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.998+0000 7f14a009c740 1 -- 172.21.15.28:0/2279198863 wait complete. 2024-09-21T05:54:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:20.999+0000 7f14a009c740 20 client.17250 trim_cache size 0 max 16384 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 Processor -- start 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 -- start start 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa42980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa33100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14941ad570 con 0x55636a66d390 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14941af1d0 con 0x55636a6651b0 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14a009c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14941adca0 con 0x55636a68e070 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57016/0 (socket says 172.21.15.28:57016) 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 -- 172.21.15.28:0/4057860548 learned_addr learned my addr 172.21.15.28:0/4057860548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14937fe640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa42980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 -- 172.21.15.28:0/4057860548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 msgr2=0x55636aa33100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1493fff640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa33100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa33100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 -- 172.21.15.28:0/4057860548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 msgr2=0x55636aa42980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa42980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f1492ffd640 1 -- 172.21.15.28:0/4057860548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636aa33e80 con 0x55636a66d390 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.012+0000 7f14937fe640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa42980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.013+0000 7f1493fff640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa33100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.013+0000 7f1492ffd640 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f148800b9d0 tx=0x7f148800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1aaddfbaafe9038 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.013+0000 7f14797fa640 1 -- 172.21.15.28:0/4057860548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488015070 con 0x55636a66d390 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.013+0000 7f14797fa640 1 -- 172.21.15.28:0/4057860548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148800b3a0 con 0x55636a66d390 2024-09-21T05:54:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.013+0000 7f14797fa640 1 -- 172.21.15.28:0/4057860548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148801aa90 con 0x55636a66d390 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- 172.21.15.28:0/4057860548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 msgr2=0x55636aa32bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f148800b9d0 tx=0x7f148800ba90 comp rx=0 tx=0).stop 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- 172.21.15.28:0/4057860548 shutdown_connections 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a68e070 0x55636aa33100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa32bc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 --2- 172.21.15.28:0/4057860548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a6651b0 0x55636aa42980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- 172.21.15.28:0/4057860548 >> 172.21.15.28:0/4057860548 conn(0x55636a682460 msgr2=0x55636aa42270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- 172.21.15.28:0/4057860548 shutdown_connections 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- 172.21.15.28:0/4057860548 wait complete. 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 -- start start 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.014+0000 7f14a009c740 1 Processor -- start 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa5a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14941aead0 con 0x55636a66d390 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14941ae3a0 con 0x55636a68e070 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14941ab550 con 0x55636a6651b0 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14a009c740 10 client.0 _ll_register_callbacks cb 0x7f148c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57022/0 (socket says 172.21.15.28:57022) 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 -- 172.21.15.28:0/2658586630 learned_addr learned my addr 172.21.15.28:0/2658586630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f14937fe640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa5a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1493fff640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 -- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 msgr2=0x55636aa5a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa5a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 -- 172.21.15.28:0/2658586630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 msgr2=0x55636aa583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 -- 172.21.15.28:0/2658586630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636aa588e0 con 0x55636a66d390 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1492ffd640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f14880047f0 tx=0x7f148801aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ec0b7b341ef22c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.015+0000 7f1472ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488015040 con 0x55636a66d390 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f148800b3a0 con 0x55636a66d390 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f1488004020 con 0x55636a66d390 2024-09-21T05:54:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1488004220 con 0x55636a66d390 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 10 client.17256 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55636aac1750 con 0x55636a66d390 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148800b3a0 con 0x55636a66d390 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f14880035f0 con 0x55636a66d390 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 10 client.17256 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 20 client.17256 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a621da0 0x55636aac3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 4 client.17256 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.016+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55636a9a98a0 con 0x55636a621da0 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f14937fe640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a621da0 0x55636aac3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f14937fe640 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a621da0 0x55636aac3e20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c5f107f18ecc495 server_cookie=108c66d0a132bc65 in_seq=0 out_seq=0 2024-09-21T05:54:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f1472ffd640 10 client.17256 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f1472ffd640 1 -- 172.21.15.28:0/2658586630 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55636a9a98a0 con 0x55636a621da0 2024-09-21T05:54:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f1472ffd640 10 client.17256 handle_command_reply: tid=0 2024-09-21T05:54:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.017+0000 7f14a009c740 1 client.17256 shutdown 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 msgr2=0x55636aa57e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f14880047f0 tx=0x7f148801aee0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 shutdown_connections 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636a621da0 0x55636aac3e20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636a68e070 0x55636aa583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636a66d390 0x55636aa57e60 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 --2- 172.21.15.28:0/2658586630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636a6651b0 0x55636aa5a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1472ffd640 0 client.17256 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 >> 172.21.15.28:0/2658586630 conn(0x55636a682460 msgr2=0x55636aa42200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 shutdown_connections 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 1 -- 172.21.15.28:0/2658586630 wait complete. 2024-09-21T05:54:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f14a009c740 20 client.17256 trim_cache size 0 max 16384 2024-09-21T05:54:21.025 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:54:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 msgr2=0x7f147c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 0x7f147c0b8b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f148c004af0 tx=0x7f148c005d80 comp rx=0 tx=0).stop 2024-09-21T05:54:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 msgr2=0x7f149419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f148400c970 tx=0x7f148400ce30 comp rx=0 tx=0).stop 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 shutdown_connections 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1494109df0 0x7f149419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14941094a0 0x7f149419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f147c0b6680 0x7f147c0b8b90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 --2- 172.21.15.28:0/2133212406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1494103b60 0x7f149419a420 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.018+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 >> 172.21.15.28:0/2133212406 conn(0x7f149407b850 msgr2=0x7f14941074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.019+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 shutdown_connections 2024-09-21T05:54:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:21.019+0000 7f1499b28640 1 -- 172.21.15.28:0/2133212406 wait complete. 2024-09-21T05:54:22.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:22 smithi028 ceph-mon[31227]: pgmap v547: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 99 op/s 2024-09-21T05:54:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:22 smithi181 ceph-mon[34819]: pgmap v547: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 99 op/s 2024-09-21T05:54:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:22 smithi129 ceph-mon[34885]: pgmap v547: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 99 op/s 2024-09-21T05:54:22.718 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:54:22.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:54:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2291915661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f81083a0 msgr2=0x7f12f81087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2291915661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f81083a0 0x7f12f81087b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009900 tx=0x7f12e802a3f0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2291915661 shutdown_connections 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2291915661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f8108cf0 0x7f12f810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2291915661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f81083a0 0x7f12f81087b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2291915661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 0x7f12f8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2291915661 >> 172.21.15.28:0/2291915661 conn(0x7f12f8079730 msgr2=0x7f12f8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2291915661 shutdown_connections 2024-09-21T05:54:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.512+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2291915661 wait complete. 2024-09-21T05:54:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 Processor -- start 2024-09-21T05:54:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 -- start start 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 0x7f12f819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f81083a0 0x7f12f819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.513+0000 7f12ffd4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8113a80 con 0x7f12f8108cf0 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12ffd4e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8113900 con 0x7f12f81083a0 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12ffd4e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8113c00 con 0x7f12f8102a60 2024-09-21T05:54:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57050/0 (socket says 172.21.15.28:57050) 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 -- 172.21.15.28:0/2495567143 learned_addr learned my addr 172.21.15.28:0/2495567143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f81083a0 0x7f12f819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 0x7f12f819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 -- 172.21.15.28:0/2495567143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 msgr2=0x7f12f819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 0x7f12f819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 -- 172.21.15.28:0/2495567143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f81083a0 msgr2=0x7f12f819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f81083a0 0x7f12f819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 -- 172.21.15.28:0/2495567143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f81a0360 con 0x7f12f8108cf0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.514+0000 7f12fe2eb640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f12f000cce0 tx=0x7f12f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.515+0000 7f12eeffd640 1 -- 172.21.15.28:0/2495567143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f000f450 con 0x7f12f8108cf0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.515+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f81a0650 con 0x7f12f8108cf0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.515+0000 7f12eeffd640 1 -- 172.21.15.28:0/2495567143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f000faa0 con 0x7f12f8108cf0 2024-09-21T05:54:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.515+0000 7f12eeffd640 1 -- 172.21.15.28:0/2495567143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0018a80 con 0x7f12f8108cf0 2024-09-21T05:54:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.515+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f81a0b10 con 0x7f12f8108cf0 2024-09-21T05:54:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.517+0000 7f13062c2740 1 Processor -- start 2024-09-21T05:54:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.517+0000 7f12eeffd640 1 -- 172.21.15.28:0/2495567143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f12f000fc50 con 0x7f12f8108cf0 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 -- start start 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b724e60 con 0x55fc7b3de490 2024-09-21T05:54:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b714290 con 0x55fc7b407350 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f13062c2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b7180a0 con 0x55fc7b3e6670 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f12fdaea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f12fdaea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57062/0 (socket says 172.21.15.28:57062) 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f12fdaea640 1 -- 172.21.15.28:0/2826459579 learned_addr learned my addr 172.21.15.28:0/2826459579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.518+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 -- 172.21.15.28:0/2826459579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 msgr2=0x55fc7b7aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12eeffd640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 0x7f12e40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 -- 172.21.15.28:0/2826459579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 msgr2=0x55fc7b7ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 -- 172.21.15.28:0/2826459579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc7b472930 con 0x55fc7b3de490 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fe2eb640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 0x7f12e40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ad00 tx=0x7f12f400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104406e803330966 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12eeffd640 1 -- 172.21.15.28:0/2495567143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f12f00e7a30 con 0x7f12f8108cf0 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12e3fff640 1 -- 172.21.15.28:0/2826459579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f401d070 con 0x55fc7b3de490 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12e3fff640 1 -- 172.21.15.28:0/2826459579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f4018af0 con 0x55fc7b3de490 2024-09-21T05:54:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.519+0000 7f12e3fff640 1 -- 172.21.15.28:0/2826459579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f4010a70 con 0x55fc7b3de490 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 -- 172.21.15.28:0/2826459579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 msgr2=0x55fc7b7ae970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f12fe2eb640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 0x7f12e40b8bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f12f00aaaf0 tx=0x7f12f00718c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ad00 tx=0x7f12f400f790 comp rx=0 tx=0).stop 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 -- 172.21.15.28:0/2826459579 shutdown_connections 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ae970 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 --2- 172.21.15.28:0/2826459579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 -- 172.21.15.28:0/2826459579 >> 172.21.15.28:0/2826459579 conn(0x55fc7b3fb740 msgr2=0x55fc7b7ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 -- 172.21.15.28:0/2826459579 shutdown_connections 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.520+0000 7f13062c2740 1 -- 172.21.15.28:0/2826459579 wait complete. 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 -- start start 2024-09-21T05:54:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 Processor -- start 2024-09-21T05:54:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b575890 con 0x55fc7b3de490 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b6aed40 con 0x55fc7b407350 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fc7b7bcc10 con 0x55fc7b3e6670 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f13062c2740 10 client.0 _ll_register_callbacks cb 0x55fc7b712c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57074/0 (socket says 172.21.15.28:57074) 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.521+0000 7f12fd2e9640 1 -- 172.21.15.28:0/2755347076 learned_addr learned my addr 172.21.15.28:0/2755347076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 -- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 msgr2=0x55fc7b7ba550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7ba550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 -- 172.21.15.28:0/2755347076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 msgr2=0x55fc7b7c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 -- 172.21.15.28:0/2755347076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc7b7c6490 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f12f806aaa0 tx=0x7f12e802a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24037e43329f709b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8041070 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f12e8004250 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12f4018af0 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f12e8051050 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f13062c2740 10 client.17268 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.522+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fc7b7c8da0 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f4018af0 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f12e80049d0 con 0x55fc7b3de490 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 10 client.17268 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 20 client.17268 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b3c5cd0 0x55fc7b7cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 4 client.17268 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc7b726db0 con 0x55fc7b3c5cd0 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.523+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b3c5cd0 0x55fc7b7cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.524+0000 7f12fdaea640 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b3c5cd0 0x55fc7b7cc340 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2acbed7ab979041b server_cookie=c7ff9f531e6ee4a1 in_seq=0 out_seq=0 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.524+0000 7f12e17fa640 10 client.17268 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f12e17fa640 1 -- 172.21.15.28:0/2755347076 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fc7b726db0 con 0x55fc7b3c5cd0 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f12e17fa640 10 client.17268 handle_command_reply: tid=0 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 client.17268 shutdown 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 msgr2=0x55fc7b7ba010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f12f806aaa0 tx=0x7f12e802a3f0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 shutdown_connections 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b3c5cd0 0x55fc7b7cc340 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 --2- 172.21.15.28:0/2755347076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7ba010 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f12e17fa640 0 client.17268 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 >> 172.21.15.28:0/2755347076 conn(0x55fc7b3fb740 msgr2=0x55fc7b7b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 shutdown_connections 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 1 -- 172.21.15.28:0/2755347076 wait complete. 2024-09-21T05:54:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.525+0000 7f13062c2740 20 client.17268 trim_cache size 0 max 16384 2024-09-21T05:54:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 Processor -- start 2024-09-21T05:54:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 -- start start 2024-09-21T05:54:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7b7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7b7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81af730 con 0x55fc7b3de490 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81b1390 con 0x55fc7b407350 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f13062c2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81afe60 con 0x55fc7b3e6670 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57078/0 (socket says 172.21.15.28:57078) 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f12fd2e9640 1 -- 172.21.15.28:0/716136300 learned_addr learned my addr 172.21.15.28:0/716136300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.543+0000 7f12fdaea640 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 -- 172.21.15.28:0/716136300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 msgr2=0x55fc7b7b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 -- 172.21.15.28:0/716136300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 msgr2=0x55fc7b7b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 -- 172.21.15.28:0/716136300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc7b7ac860 con 0x55fc7b3de490 2024-09-21T05:54:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f12e8033000 tx=0x7f12e8003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bb3e74a116fd67 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12e37fe640 1 -- 172.21.15.28:0/716136300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8041040 con 0x55fc7b3de490 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12e37fe640 1 -- 172.21.15.28:0/716136300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e8005420 con 0x55fc7b3de490 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.544+0000 7f12e37fe640 1 -- 172.21.15.28:0/716136300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8035070 con 0x55fc7b3de490 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- 172.21.15.28:0/716136300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 msgr2=0x55fc7b7bba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f12e8033000 tx=0x7f12e8003780 comp rx=0 tx=0).stop 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- 172.21.15.28:0/716136300 shutdown_connections 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b407350 0x55fc7b7b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b3e6670 0x55fc7b7b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 --2- 172.21.15.28:0/716136300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7bba40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- 172.21.15.28:0/716136300 >> 172.21.15.28:0/716136300 conn(0x55fc7b3fb740 msgr2=0x55fc7b7bb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- 172.21.15.28:0/716136300 shutdown_connections 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- 172.21.15.28:0/716136300 wait complete. 2024-09-21T05:54:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 -- start start 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.545+0000 7f13062c2740 1 Processor -- start 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 0x55fc7b7bed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b407350 0x55fc7b7bf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81b0c90 con 0x55fc7b3de490 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81b0560 con 0x55fc7b3e6670 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81ad710 con 0x55fc7b407350 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f13062c2740 10 client.0 _ll_register_callbacks cb 0x7f12e8031c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57088/0 (socket says 172.21.15.28:57088) 2024-09-21T05:54:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 -- 172.21.15.28:0/254658196 learned_addr learned my addr 172.21.15.28:0/254658196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b407350 0x55fc7b7bf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fdaea640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 0x55fc7b7bed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 -- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b407350 msgr2=0x55fc7b7bf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b407350 0x55fc7b7bf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 -- 172.21.15.28:0/254658196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 msgr2=0x55fc7b7bed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 0x55fc7b7bed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fd2e9640 1 -- 172.21.15.28:0/254658196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc7b836d50 con 0x55fc7b3de490 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.546+0000 7f12fdaea640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 0x55fc7b7bed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12fd2e9640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f12e8031a40 tx=0x7f12e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0860c777f37738 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8035030 con 0x55fc7b3de490 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f12e8004f70 con 0x55fc7b3de490 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e8005420 con 0x55fc7b3de490 2024-09-21T05:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f12e8041070 con 0x55fc7b3de490 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f13062c2740 10 client.17274 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fc7b837ee0 con 0x55fc7b3de490 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e8005420 con 0x55fc7b3de490 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.547+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f12e8032730 con 0x55fc7b3de490 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 10 client.17274 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 10 client.17274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 20 client.17274 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b39b080 0x55fc7b83a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 4 client.17274 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fc7b722860 con 0x55fc7b39b080 2024-09-21T05:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f12fdaea640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b39b080 0x55fc7b83a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f12fdaea640 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b39b080 0x55fc7b83a6a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3d09a1eb92d2f99 server_cookie=151c4dec960ee32b in_seq=0 out_seq=0 2024-09-21T05:54:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.548+0000 7f12e17fa640 10 client.17274 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f12e17fa640 1 -- 172.21.15.28:0/254658196 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fc7b722860 con 0x55fc7b39b080 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f12e17fa640 10 client.17274 handle_command_reply: tid=0 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 client.17274 shutdown 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 msgr2=0x55fc7b7c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f12e8031a40 tx=0x7f12e80057f0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 shutdown_connections 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55fc7b39b080 0x55fc7b83a6a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fc7b407350 0x55fc7b7bf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fc7b3e6670 0x55fc7b7bed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f12e17fa640 0 client.17274 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 --2- 172.21.15.28:0/254658196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc7b3de490 0x55fc7b7c2860 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 >> 172.21.15.28:0/254658196 conn(0x55fc7b3fb740 msgr2=0x55fc7b7bb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 shutdown_connections 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 1 -- 172.21.15.28:0/254658196 wait complete. 2024-09-21T05:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.550+0000 7f13062c2740 20 client.17274 trim_cache size 0 max 16384 2024-09-21T05:54:23.556 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:54:23.556 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:54:23.556 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88", 2024-09-21T05:54:23.556 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:54:23.556 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 msgr2=0x7f12e40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 0x7f12e40b8bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f12f00aaaf0 tx=0x7f12f00718c0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 msgr2=0x7f12f819fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f12f000cce0 tx=0x7f12f000a620 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 shutdown_connections 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f12e40b66b0 0x7f12e40b8bc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f8108cf0 0x7f12f819fcf0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12f81083a0 0x7f12f819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 --2- 172.21.15.28:0/2495567143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12f8102a60 0x7f12f819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:23.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 >> 172.21.15.28:0/2495567143 conn(0x7f12f8079730 msgr2=0x7f12f8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:23.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 shutdown_connections 2024-09-21T05:54:23.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:23.551+0000 7f12ffd4e640 1 -- 172.21.15.28:0/2495567143 wait complete. 2024-09-21T05:54:24.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:54:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:24 smithi181 ceph-mon[34819]: pgmap v548: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 85 op/s 2024-09-21T05:54:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:24 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:54:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:24 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:24 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:54:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:24 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:24.558 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:24 smithi028 ceph-mon[31227]: pgmap v548: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 85 op/s 2024-09-21T05:54:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:24 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:54:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:24 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:24 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:54:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:24 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:24 smithi129 ceph-mon[34885]: pgmap v548: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 85 op/s 2024-09-21T05:54:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:24 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:54:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:24 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:24 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:54:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:24 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:54:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 -- 172.21.15.28:0/1401269417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341083a0 msgr2=0x7ff4341087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 --2- 172.21.15.28:0/1401269417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341083a0 0x7ff4341087b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff428009900 tx=0x7ff42802a410 comp rx=0 tx=0).stop 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 -- 172.21.15.28:0/1401269417 shutdown_connections 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 --2- 172.21.15.28:0/1401269417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff434108cf0 0x7ff43410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 --2- 172.21.15.28:0/1401269417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341083a0 0x7ff4341087b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 --2- 172.21.15.28:0/1401269417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434102a60 0x7ff434102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 -- 172.21.15.28:0/1401269417 >> 172.21.15.28:0/1401269417 conn(0x7ff434079730 msgr2=0x7ff434079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 -- 172.21.15.28:0/1401269417 shutdown_connections 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.042+0000 7ff43a674640 1 -- 172.21.15.28:0/1401269417 wait complete. 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 Processor -- start 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 -- start start 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4341083a0 0x7ff434199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434108cf0 0x7ff434199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341129a0 con 0x7ff434102a60 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112820 con 0x7ff434108cf0 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.043+0000 7ff43a674640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112b20 con 0x7ff4341083a0 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57114/0 (socket says 172.21.15.28:57114) 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 -- 172.21.15.28:0/214139514 learned_addr learned my addr 172.21.15.28:0/214139514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff438c11640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434108cf0 0x7ff434199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 -- 172.21.15.28:0/214139514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4341083a0 msgr2=0x7ff434199640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4341083a0 0x7ff434199640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 -- 172.21.15.28:0/214139514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434108cf0 msgr2=0x7ff434199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434108cf0 0x7ff434199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 -- 172.21.15.28:0/214139514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4341a87b0 con 0x7ff434102a60 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff433fff640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff42400d950 tx=0x7ff42400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff4317fa640 1 -- 172.21.15.28:0/214139514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424017030 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff4317fa640 1 -- 172.21.15.28:0/214139514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff424004020 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341a8aa0 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.044+0000 7ff4317fa640 1 -- 172.21.15.28:0/214139514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240047a0 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.045+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a9030 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 Processor -- start 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 -- start start 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff4317fa640 1 -- 172.21.15.28:0/214139514 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff424009a30 con 0x7ff434102a60 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d28270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d287b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d481c9ee50 con 0x55d481960390 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d481c8e280 con 0x55d481981070 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.047+0000 7ff440be8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d481c92090 con 0x55d4819581b0 2024-09-21T05:54:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57122/0 (socket says 172.21.15.28:57122) 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 -- 172.21.15.28:0/1533793270 learned_addr learned my addr 172.21.15.28:0/1533793270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff433fff640 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d287b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff438c11640 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d28270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 -- 172.21.15.28:0/1533793270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 msgr2=0x55d481d287b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d287b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 -- 172.21.15.28:0/1533793270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 msgr2=0x55d481d28270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d28270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 -- 172.21.15.28:0/1533793270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4819ec650 con 0x55d481960390 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4317fa640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 0x7ff41c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4337fe640 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff428006010 tx=0x7ff428005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621ea2968a0fafe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff438c11640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 0x7ff41c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff4317fa640 1 -- 172.21.15.28:0/214139514 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7ff4240e7270 con 0x7ff434102a60 2024-09-21T05:54:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.048+0000 7ff41a7fc640 1 -- 172.21.15.28:0/1533793270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42803e030 con 0x55d481960390 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff41a7fc640 1 -- 172.21.15.28:0/1533793270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff428032d60 con 0x55d481960390 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff41a7fc640 1 -- 172.21.15.28:0/1533793270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428035040 con 0x55d481960390 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 -- 172.21.15.28:0/1533793270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 msgr2=0x55d481d28cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff438c11640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 0x7ff41c0b8c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c00e040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff428006010 tx=0x7ff428005f00 comp rx=0 tx=0).stop 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 -- 172.21.15.28:0/1533793270 shutdown_connections 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d28cf0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d287b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.049+0000 7ff440be8740 1 --2- 172.21.15.28:0/1533793270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d28270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.050+0000 7ff440be8740 1 -- 172.21.15.28:0/1533793270 >> 172.21.15.28:0/1533793270 conn(0x55d481975460 msgr2=0x55d481d26160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.050+0000 7ff440be8740 1 -- 172.21.15.28:0/1533793270 shutdown_connections 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.050+0000 7ff440be8740 1 -- 172.21.15.28:0/1533793270 wait complete. 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.050+0000 7ff440be8740 1 -- start start 2024-09-21T05:54:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.050+0000 7ff440be8740 1 Processor -- start 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 0x55d481d33de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d481aef700 con 0x55d4819581b0 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d481c29110 con 0x55d481981070 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d481d369f0 con 0x55d481960390 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 10 client.0 _ll_register_callbacks cb 0x55d481c42990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57138/0 (socket says 172.21.15.28:57138) 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 -- 172.21.15.28:0/817008184 learned_addr learned my addr 172.21.15.28:0/817008184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff438c11640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff4337fe640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 0x55d481d33de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 -- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 msgr2=0x55d481d33de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 0x55d481d33de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 -- 172.21.15.28:0/817008184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 msgr2=0x55d481d34360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d34360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 -- 172.21.15.28:0/817008184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d481c56c40 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff438c11640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d34360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff4337fe640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 0x55d481d33de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff433fff640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff4240aa330 tx=0x7ff4240ea950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdfd00e162af488 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff4137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240a8f60 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7ff4240a8a10 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.051+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff428032d60 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff4240ac830 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 10 client.17286 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d481d42720 con 0x55d4819581b0 2024-09-21T05:54:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff428032d60 con 0x55d4819581b0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff4240a8c10 con 0x55d4819581b0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 10 client.17286 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 20 client.17286 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d48193f9f0 0x55d481d45c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 4 client.17286 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d481ca0da0 con 0x55d48193f9f0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.052+0000 7ff4337fe640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d48193f9f0 0x55d481d45c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.053+0000 7ff4337fe640 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d48193f9f0 0x55d481d45c90 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b965ed10c16f1f server_cookie=db7cf4f8637eb34 in_seq=0 out_seq=0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.053+0000 7ff4137fe640 10 client.17286 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.053+0000 7ff4137fe640 1 -- 172.21.15.28:0/817008184 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d481ca0da0 con 0x55d48193f9f0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.053+0000 7ff4137fe640 10 client.17286 handle_command_reply: tid=0 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 client.17286 shutdown 2024-09-21T05:54:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 msgr2=0x55d481d338a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88", 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:54:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff4240aa330 tx=0x7ff4240ea950 comp rx=0 tx=0).stop 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 shutdown_connections 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d48193f9f0 0x55d481d45c90 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d34360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d481960390 0x55d481d33de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff4137fe640 0 client.17286 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 --2- 172.21.15.28:0/817008184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4819581b0 0x55d481d338a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 >> 172.21.15.28:0/817008184 conn(0x55d481975460 msgr2=0x55d481d2b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 shutdown_connections 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 1 -- 172.21.15.28:0/817008184 wait complete. 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.054+0000 7ff440be8740 20 client.17286 trim_cache size 0 max 16384 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 Processor -- start 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 -- start start 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d29d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d30a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b1900 con 0x55d481960390 2024-09-21T05:54:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b3560 con 0x55d481981070 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff440be8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b2030 con 0x55d4819581b0 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57154/0 (socket says 172.21.15.28:57154) 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 -- 172.21.15.28:0/2854590710 learned_addr learned my addr 172.21.15.28:0/2854590710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff433fff640 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d29d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 -- 172.21.15.28:0/2854590710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 msgr2=0x55d481d29d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d29d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 -- 172.21.15.28:0/2854590710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 msgr2=0x55d481d30a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d30a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.072+0000 7ff4337fe640 1 -- 172.21.15.28:0/2854590710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d481d266a0 con 0x55d481960390 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.073+0000 7ff4337fe640 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff428000c00 tx=0x7ff428032f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0a4ad0df470c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.073+0000 7ff419ffb640 1 -- 172.21.15.28:0/2854590710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42803e040 con 0x55d481960390 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.073+0000 7ff419ffb640 1 -- 172.21.15.28:0/2854590710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff428032d60 con 0x55d481960390 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.073+0000 7ff419ffb640 1 -- 172.21.15.28:0/2854590710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428035040 con 0x55d481960390 2024-09-21T05:54:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- 172.21.15.28:0/2854590710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 msgr2=0x55d481d354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff428000c00 tx=0x7ff428032f10 comp rx=0 tx=0).stop 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- 172.21.15.28:0/2854590710 shutdown_connections 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d30a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d354d0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 --2- 172.21.15.28:0/2854590710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d29d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- 172.21.15.28:0/2854590710 >> 172.21.15.28:0/2854590710 conn(0x55d481975460 msgr2=0x55d481d34dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- 172.21.15.28:0/2854590710 shutdown_connections 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- 172.21.15.28:0/2854590710 wait complete. 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 -- start start 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.074+0000 7ff440be8740 1 Processor -- start 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d37da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d38820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b2e60 con 0x55d481960390 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b2730 con 0x55d481981070 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341af8e0 con 0x55d4819581b0 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff440be8740 10 client.0 _ll_register_callbacks cb 0x7ff4240046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57170/0 (socket says 172.21.15.28:57170) 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 -- 172.21.15.28:0/208810135 learned_addr learned my addr 172.21.15.28:0/208810135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff438c11640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d38820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff433fff640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d37da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 -- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 msgr2=0x55d481d37da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d37da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 -- 172.21.15.28:0/208810135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 msgr2=0x55d481d38820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d38820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 -- 172.21.15.28:0/208810135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d481daf7f0 con 0x55d481960390 2024-09-21T05:54:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff438c11640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d38820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4337fe640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff428031440 tx=0x7ff428004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d047186e978390 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.075+0000 7ff4137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428058070 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7ff428031860 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff428032d60 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff428064050 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 10 client.17292 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d481db0a80 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff428032d60 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff428003d50 con 0x55d481960390 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 10 client.17292 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 10 client.17292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 20 client.17292 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d481914da0 0x55d481db2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 4 client.17292 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.076+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d481c9c850 con 0x55d481914da0 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.077+0000 7ff433fff640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d481914da0 0x55d481db2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.077+0000 7ff433fff640 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d481914da0 0x55d481db2d60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b7e89cc9318cea6 server_cookie=9ca60597bcdcd22a in_seq=0 out_seq=0 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.077+0000 7ff4137fe640 10 client.17292 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff4137fe640 1 -- 172.21.15.28:0/208810135 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3410 (crc 0 0 0) 0x7ff424014d60 con 0x55d481914da0 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff4137fe640 10 client.17292 handle_command_reply: tid=0 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 client.17292 shutdown 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 msgr2=0x55d481d382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff428031440 tx=0x7ff428004740 comp rx=0 tx=0).stop 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 shutdown_connections 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d481914da0 0x55d481db2d60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d481981070 0x55d481d38820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d481960390 0x55d481d382e0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff4137fe640 0 client.17292 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 --2- 172.21.15.28:0/208810135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d4819581b0 0x55d481d37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 >> 172.21.15.28:0/208810135 conn(0x55d481975460 msgr2=0x55d481d34d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 shutdown_connections 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 1 -- 172.21.15.28:0/208810135 wait complete. 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.078+0000 7ff440be8740 20 client.17292 trim_cache size 0 max 16384 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 msgr2=0x7ff41c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 0x7ff41c0b8c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c00e040 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 msgr2=0x7ff434199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff42400d950 tx=0x7ff42400de10 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 shutdown_connections 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff434108cf0 0x7ff434199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4341083a0 0x7ff434199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff41c0b6700 0x7ff41c0b8c10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 --2- 172.21.15.28:0/214139514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff434102a60 0x7ff434199100 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 >> 172.21.15.28:0/214139514 conn(0x7ff434079730 msgr2=0x7ff4341052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 shutdown_connections 2024-09-21T05:54:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:25.079+0000 7ff43a674640 1 -- 172.21.15.28:0/214139514 wait complete. 2024-09-21T05:54:26.261 INFO:tasks.cephfs.filesystem:scrub status for tag:cc0adacb-ac45-4c09-9c6d-8bf3a46edf88 - {'path': '/', 'tag': 'cc0adacb-ac45-4c09-9c6d-8bf3a46edf88', 'options': 'recursive,force'} 2024-09-21T05:54:26.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:26 smithi181 ceph-mon[34819]: pgmap v549: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 53 op/s 2024-09-21T05:54:26.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:26 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:54:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:26 smithi028 ceph-mon[31227]: pgmap v549: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 53 op/s 2024-09-21T05:54:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:26 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:54:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:26 smithi129 ceph-mon[34885]: pgmap v549: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 53 op/s 2024-09-21T05:54:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:26 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:54:28.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:28 smithi181 ceph-mon[34819]: pgmap v550: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.1 KiB/s wr, 23 op/s 2024-09-21T05:54:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:28 smithi028 ceph-mon[31227]: pgmap v550: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.1 KiB/s wr, 23 op/s 2024-09-21T05:54:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:28 smithi129 ceph-mon[34885]: pgmap v550: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.1 KiB/s wr, 23 op/s 2024-09-21T05:54:30.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:30 smithi181 ceph-mon[34819]: pgmap v551: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.9 KiB/s wr, 20 op/s 2024-09-21T05:54:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:30 smithi028 ceph-mon[31227]: pgmap v551: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.9 KiB/s wr, 20 op/s 2024-09-21T05:54:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:30 smithi129 ceph-mon[34885]: pgmap v551: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.9 KiB/s wr, 20 op/s 2024-09-21T05:54:32.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:32 smithi181 ceph-mon[34819]: pgmap v552: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T05:54:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:32 smithi028 ceph-mon[31227]: pgmap v552: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T05:54:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:32 smithi129 ceph-mon[34885]: pgmap v552: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T05:54:34.545 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:34 smithi181 ceph-mon[34819]: pgmap v553: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T05:54:34.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:34 smithi028 ceph-mon[31227]: pgmap v553: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T05:54:34.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:34 smithi129 ceph-mon[34885]: pgmap v553: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T05:54:36.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:36 smithi181 ceph-mon[34819]: pgmap v554: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 11 op/s 2024-09-21T05:54:36.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:36 smithi028 ceph-mon[31227]: pgmap v554: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 11 op/s 2024-09-21T05:54:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:36 smithi129 ceph-mon[34885]: pgmap v554: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 11 op/s 2024-09-21T05:54:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:38 smithi181 ceph-mon[34819]: pgmap v555: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T05:54:38.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:38 smithi028 ceph-mon[31227]: pgmap v555: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T05:54:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:38 smithi129 ceph-mon[34885]: pgmap v555: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T05:54:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:40 smithi181 ceph-mon[34819]: pgmap v556: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-09-21T05:54:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:40 smithi028 ceph-mon[31227]: pgmap v556: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-09-21T05:54:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:40 smithi129 ceph-mon[34885]: pgmap v556: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-09-21T05:54:42.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:42 smithi181 ceph-mon[34819]: pgmap v557: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:42.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:42 smithi028 ceph-mon[31227]: pgmap v557: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:42 smithi129 ceph-mon[34885]: pgmap v557: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:44.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:44 smithi181 ceph-mon[34819]: pgmap v558: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:44 smithi028 ceph-mon[31227]: pgmap v558: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:44 smithi129 ceph-mon[34885]: pgmap v558: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:46.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:46 smithi181 ceph-mon[34819]: pgmap v559: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:46 smithi028 ceph-mon[31227]: pgmap v559: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:46 smithi129 ceph-mon[34885]: pgmap v559: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:48.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:48 smithi181 ceph-mon[34819]: pgmap v560: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:48 smithi028 ceph-mon[31227]: pgmap v560: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:48 smithi129 ceph-mon[34885]: pgmap v560: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:50.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:50 smithi181 ceph-mon[34819]: pgmap v561: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:54:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:50 smithi028 ceph-mon[31227]: pgmap v561: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:54:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:50 smithi129 ceph-mon[34885]: pgmap v561: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:54:52.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:52 smithi181 ceph-mon[34819]: pgmap v562: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:52 smithi028 ceph-mon[31227]: pgmap v562: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:52 smithi129 ceph-mon[34885]: pgmap v562: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:54:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:54 smithi181 ceph-mon[34819]: pgmap v563: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:54:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:54:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:54:54.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:54:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:54 smithi028 ceph-mon[31227]: pgmap v563: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:54:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:54:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:54:54.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:54:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:54 smithi129 ceph-mon[34885]: pgmap v563: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:54:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:54:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:54:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:54:54.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:54:56.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:54:56.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:56.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:56 smithi181 ceph-mon[34819]: pgmap v564: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:56.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:56 smithi028 ceph-mon[31227]: pgmap v564: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:56 smithi129 ceph-mon[34885]: pgmap v564: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:54:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 -- 172.21.15.28:0/3181208297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc109df0 msgr2=0x7fdfdc110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/3181208297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc109df0 0x7fdfdc110620 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009900 tx=0x7fdfd402a410 comp rx=0 tx=0).stop 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 -- 172.21.15.28:0/3181208297 shutdown_connections 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/3181208297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc109df0 0x7fdfdc110620 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/3181208297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc1094a0 0x7fdfdc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/3181208297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc103b60 0x7fdfdc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 -- 172.21.15.28:0/3181208297 >> 172.21.15.28:0/3181208297 conn(0x7fdfdc07b850 msgr2=0x7fdfdc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 -- 172.21.15.28:0/3181208297 shutdown_connections 2024-09-21T05:54:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.962+0000 7fdfe2d49640 1 -- 172.21.15.28:0/3181208297 wait complete. 2024-09-21T05:54:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 Processor -- start 2024-09-21T05:54:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 -- start start 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc1094a0 0x7fdfdc19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc109df0 0x7fdfdc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc113b10 con 0x7fdfdc103b60 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc113990 con 0x7fdfdc109df0 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.963+0000 7fdfe2d49640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc113c90 con 0x7fdfdc1094a0 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49818/0 (socket says 172.21.15.28:49818) 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/635635569 learned_addr learned my addr 172.21.15.28:0/635635569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc109df0 0x7fdfdc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc1094a0 0x7fdfdc19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/635635569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc1094a0 msgr2=0x7fdfdc19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc1094a0 0x7fdfdc19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/635635569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc109df0 msgr2=0x7fdfdc19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc109df0 0x7fdfdc19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/635635569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc1a03f0 con 0x7fdfdc103b60 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.964+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400b4a0 tx=0x7fdfc400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.965+0000 7fdfd9ffb640 1 -- 172.21.15.28:0/635635569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4014070 con 0x7fdfdc103b60 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.965+0000 7fdfd9ffb640 1 -- 172.21.15.28:0/635635569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfc4010ac0 con 0x7fdfdc103b60 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.965+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc1a06e0 con 0x7fdfdc103b60 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.965+0000 7fdfd9ffb640 1 -- 172.21.15.28:0/635635569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4017c30 con 0x7fdfdc103b60 2024-09-21T05:54:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.965+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc1a6e40 con 0x7fdfdc103b60 2024-09-21T05:54:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 Processor -- start 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 -- start start 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfd9ffb640 1 -- 172.21.15.28:0/635635569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fdfc4017070 con 0x7fdfdc103b60 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 0x5649087f8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908430390 0x5649087f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56490876ee50 con 0x5649084281b0 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56490875e280 con 0x564908451070 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564908762090 con 0x564908430390 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49832/0 (socket says 172.21.15.28:49832) 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/4058180190 learned_addr learned my addr 172.21.15.28:0/4058180190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 0x5649087f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.968+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908430390 0x5649087f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/4058180190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908430390 msgr2=0x5649087f90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908430390 0x5649087f90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/4058180190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 msgr2=0x5649087f8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 0x5649087f8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/4058180190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649084bc650 con 0x5649084281b0 2024-09-21T05:54:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 0x5649087f8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40e4330 tx=0x7fdfc40eaf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7648df19fb3c8d17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfd9ffb640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 0x7fdfc80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfd2ffd640 1 -- 172.21.15.28:0/4058180190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40a91f0 con 0x5649084281b0 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 0x7fdfc80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfd9ffb640 1 -- 172.21.15.28:0/635635569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fdfc40e6ca0 con 0x7fdfdc103b60 2024-09-21T05:54:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfd2ffd640 1 -- 172.21.15.28:0/4058180190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfc40f4070 con 0x5649084281b0 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.969+0000 7fdfd2ffd640 1 -- 172.21.15.28:0/4058180190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40ac310 con 0x5649084281b0 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.970+0000 7fdfe92bd740 1 -- 172.21.15.28:0/4058180190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 msgr2=0x5649087f8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.970+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 0x7fdfc80b8b90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4006010 tx=0x7fdfd40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.970+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40e4330 tx=0x7fdfc40eaf10 comp rx=0 tx=0).stop 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 -- 172.21.15.28:0/4058180190 shutdown_connections 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908430390 0x5649087f90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649087f8b90 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/4058180190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908451070 0x5649087f8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 -- 172.21.15.28:0/4058180190 >> 172.21.15.28:0/4058180190 conn(0x564908445460 msgr2=0x5649087f6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 -- 172.21.15.28:0/4058180190 shutdown_connections 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 -- 172.21.15.28:0/4058180190 wait complete. 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 -- start start 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.971+0000 7fdfe92bd740 1 Processor -- start 2024-09-21T05:54:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 0x5649088080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649088144c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649085bf700 con 0x564908430390 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649086f9110 con 0x5649084281b0 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56490880b190 con 0x564908451070 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe92bd740 10 client.0 _ll_register_callbacks cb 0x564908712990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49834/0 (socket says 172.21.15.28:49834) 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 -- 172.21.15.28:0/3605179690 learned_addr learned my addr 172.21.15.28:0/3605179690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 0x5649088080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649088144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 -- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 msgr2=0x5649088144c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649088144c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 -- 172.21.15.28:0/3605179690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 msgr2=0x5649088080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 0x5649088080f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfdbfff640 1 -- 172.21.15.28:0/3605179690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564908726c40 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.972+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 0x5649088080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc004070 tx=0x7fdfcc004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139480cd26d677fe server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc004d20 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fdfcc013a40 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc40f4070 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fdfcc013c80 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfe92bd740 10 client.17304 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564908816ee0 con 0x564908430390 2024-09-21T05:54:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfc40f4070 con 0x564908430390 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.973+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fdfcc01a070 con 0x564908430390 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 10 client.17304 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 20 client.17304 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56490840f9f0 0x56490881a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 4 client.17304 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564908770da0 con 0x56490840f9f0 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56490840f9f0 0x56490881a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56490840f9f0 0x56490881a4a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13b8de564d36de63 server_cookie=5d0507e6f680e3c9 in_seq=0 out_seq=0 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.974+0000 7fdfbbfff640 10 client.17304 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.975+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3605179690 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564908770da0 con 0x56490840f9f0 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.975+0000 7fdfbbfff640 10 client.17304 handle_command_reply: tid=0 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.975+0000 7fdfe92bd740 1 client.17304 shutdown 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 msgr2=0x564908808630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc004070 tx=0x7fdfcc004530 comp rx=0 tx=0).stop 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 shutdown_connections 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x56490840f9f0 0x56490881a4a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649088144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564908430390 0x564908808630 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3605179690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649084281b0 0x5649088080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 >> 172.21.15.28:0/3605179690 conn(0x564908445460 msgr2=0x5649087f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfbbfff640 0 client.17304 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 shutdown_connections 2024-09-21T05:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3605179690 wait complete. 2024-09-21T05:54:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:56.976+0000 7fdfe92bd740 20 client.17304 trim_cache size 0 max 16384 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.007+0000 7fdfe92bd740 1 Processor -- start 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.007+0000 7fdfe92bd740 1 -- start start 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x5649088143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649087fd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ad690 con 0x5649084281b0 2024-09-21T05:54:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1af2f0 con 0x564908430390 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1addc0 con 0x564908451070 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49848/0 (socket says 172.21.15.28:49848) 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/2075809336 learned_addr learned my addr 172.21.15.28:0/2075809336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x5649088143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649087fd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/2075809336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 msgr2=0x5649087fd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649087fd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/2075809336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 msgr2=0x5649088143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x5649088143c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/2075809336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564908807050 con 0x5649084281b0 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.008+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x5649088143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.009+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40ac6b0 tx=0x7fdfc40ea790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941f48f653639e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.009+0000 7fdfd27fc640 1 -- 172.21.15.28:0/2075809336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc401e040 con 0x5649084281b0 2024-09-21T05:54:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.009+0000 7fdfd27fc640 1 -- 172.21.15.28:0/2075809336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfc40f4040 con 0x5649084281b0 2024-09-21T05:54:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.009+0000 7fdfd27fc640 1 -- 172.21.15.28:0/2075809336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40ac4f0 con 0x5649084281b0 2024-09-21T05:54:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- 172.21.15.28:0/2075809336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 msgr2=0x5649088085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40ac6b0 tx=0x7fdfc40ea790 comp rx=0 tx=0).stop 2024-09-21T05:54:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- 172.21.15.28:0/2075809336 shutdown_connections 2024-09-21T05:54:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x5649087fd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x5649088143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/2075809336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x5649088085b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- 172.21.15.28:0/2075809336 >> 172.21.15.28:0/2075809336 conn(0x564908445460 msgr2=0x564908807ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- 172.21.15.28:0/2075809336 shutdown_connections 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- 172.21.15.28:0/2075809336 wait complete. 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 -- start start 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.010+0000 7fdfe92bd740 1 Processor -- start 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x56490880c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x56490880c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1aebf0 con 0x5649084281b0 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ae4c0 con 0x564908430390 2024-09-21T05:54:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ab670 con 0x564908451070 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe92bd740 10 client.0 _ll_register_callbacks cb 0x7fdfcc009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49850/0 (socket says 172.21.15.28:49850) 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/3166829158 learned_addr learned my addr 172.21.15.28:0/3166829158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x56490880c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe12e6640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x56490880c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 msgr2=0x56490880c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x56490880c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/3166829158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 msgr2=0x56490880c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x56490880c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfe0ae5640 1 -- 172.21.15.28:0/3166829158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56490880cdc0 con 0x5649084281b0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.011+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x56490880c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe0ae5640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40ef860 tx=0x7fdfc40e7010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cb4297edd7d2ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc401e070 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc40f4040 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fdfc40d00b0 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fdfc40ac6e0 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 10 client.17310 fetch_fsmap learned FSMap version 47 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564908884dd0 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfc40f4040 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fdfc40a8e10 con 0x5649084281b0 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 10 client.17310 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 20 client.17310 populate_metadata read hostname 'smithi028' 2024-09-21T05:54:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649083e4da0 0x564908886fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 4 client.17310 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.012+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56490876c850 con 0x5649083e4da0 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649083e4da0 0x564908886fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfdbfff640 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649083e4da0 0x564908886fb0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ce282262235398 server_cookie=c86c813775801e9e in_seq=0 out_seq=0 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfbbfff640 10 client.17310 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3166829158 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3214 (crc 0 0 0) 0x56490876c850 con 0x5649083e4da0 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfbbfff640 10 client.17310 handle_command_reply: tid=0 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.013+0000 7fdfe92bd740 1 client.17310 shutdown 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 msgr2=0x564908810c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40ef860 tx=0x7fdfc40e7010 comp rx=0 tx=0).stop 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 shutdown_connections 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5649083e4da0 0x564908886fb0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564908451070 0x56490880c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564908430390 0x56490880c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfbbfff640 0 client.17310 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 --2- 172.21.15.28:0/3166829158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649084281b0 0x564908810c20 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 >> 172.21.15.28:0/3166829158 conn(0x564908445460 msgr2=0x5649087f5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 shutdown_connections 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 1 -- 172.21.15.28:0/3166829158 wait complete. 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.014+0000 7fdfe92bd740 20 client.17310 trim_cache size 0 max 16384 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:54:57.022 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.023 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:54:57.024 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.026 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:54:57.028 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 msgr2=0x7fdfc80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:57.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 0x7fdfc80b8b90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4006010 tx=0x7fdfd40058d0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 msgr2=0x7fdfdc19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400b4a0 tx=0x7fdfc400b960 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 shutdown_connections 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfdc109df0 0x7fdfdc19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfdc1094a0 0x7fdfdc19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfc80b6680 0x7fdfc80b8b90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 --2- 172.21.15.28:0/635635569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfdc103b60 0x7fdfdc19a480 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 >> 172.21.15.28:0/635635569 conn(0x7fdfdc07b850 msgr2=0x7fdfdc1074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 shutdown_connections 2024-09-21T05:54:57.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:57.015+0000 7fdfe2d49640 1 -- 172.21.15.28:0/635635569 wait complete. 2024-09-21T05:54:57.818 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:54:57.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:54:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:54:58.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:58 smithi181 ceph-mon[34819]: pgmap v565: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:58.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:58 smithi028 ceph-mon[31227]: pgmap v565: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:58 smithi129 ceph-mon[34885]: pgmap v565: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:54:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 -- 172.21.15.28:0/125780061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c1101a0 msgr2=0x7fcd7c1125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 --2- 172.21.15.28:0/125780061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1125c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009ca0 tx=0x7fcd6c02a8c0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 -- 172.21.15.28:0/125780061 shutdown_connections 2024-09-21T05:54:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 --2- 172.21.15.28:0/125780061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1125c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 --2- 172.21.15.28:0/125780061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c076080 0x7fcd7c10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 --2- 172.21.15.28:0/125780061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c075730 0x7fcd7c075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 -- 172.21.15.28:0/125780061 >> 172.21.15.28:0/125780061 conn(0x7fcd7c07b910 msgr2=0x7fcd7c07bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 -- 172.21.15.28:0/125780061 shutdown_connections 2024-09-21T05:54:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.676+0000 7fcd83dae640 1 -- 172.21.15.28:0/125780061 wait complete. 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 Processor -- start 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 -- start start 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 0x7fcd7c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c115c90 con 0x7fcd7c076080 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c115b10 con 0x7fcd7c075730 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd83dae640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c115e10 con 0x7fcd7c1101a0 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd81349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.677+0000 7fcd81349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49868/0 (socket says 172.21.15.28:49868) 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 -- 172.21.15.28:0/3339271910 learned_addr learned my addr 172.21.15.28:0/3339271910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:54:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81b4a640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 0x7fcd7c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd8234b640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 -- 172.21.15.28:0/3339271910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 msgr2=0x7fcd7c1a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 -- 172.21.15.28:0/3339271910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 msgr2=0x7fcd7c19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 0x7fcd7c19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 -- 172.21.15.28:0/3339271910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c1a2530 con 0x7fcd7c076080 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81b4a640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 0x7fcd7c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd8234b640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd81349640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d950 tx=0x7fcd7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70017070 con 0x7fcd7c076080 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c1a2820 con 0x7fcd7c076080 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd70009c30 con 0x7fcd7c076080 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd700047a0 con 0x7fcd7c076080 2024-09-21T05:54:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.678+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c1a9040 con 0x7fcd7c076080 2024-09-21T05:54:58.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.679+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7c075b40 con 0x7fcd7c076080 2024-09-21T05:54:58.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.681+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fcd70004020 con 0x7fcd7c076080 2024-09-21T05:54:58.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.682+0000 7fcd7affd640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 0x7fcd640b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:54:58.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.682+0000 7fcd81b4a640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 0x7fcd640b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:54:58.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.682+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fcd700e7860 con 0x7fcd7c076080 2024-09-21T05:54:58.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.683+0000 7fcd81b4a640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 0x7fcd640b8bc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400abe0 tx=0x7fcd7400a5c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:54:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.685+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcd70020020 con 0x7fcd7c076080 2024-09-21T05:54:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.911+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd7c19d7b0 con 0x7fcd7c076080 2024-09-21T05:54:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.912+0000 7fcd7affd640 1 -- 172.21.15.28:0/3339271910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11995 (secure 0 0 0) 0x7fcd700add70 con 0x7fcd7c076080 2024-09-21T05:54:58.914 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:54:58.914 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"btime":"2024-09-21T05:54:11:183838+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:54:11.055265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0,4],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24688,24680,24673,24664,24653]},"id":1}]} 2024-09-21T05:54:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-09-21T05:54:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 msgr2=0x7fcd640b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 0x7fcd640b8bc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400abe0 tx=0x7fcd7400a5c0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 msgr2=0x7fcd7c19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d950 tx=0x7fcd7000de10 comp rx=0 tx=0).stop 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 shutdown_connections 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd7c1101a0 0x7fcd7c1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcd640b66b0 0x7fcd640b8bc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c076080 0x7fcd7c19cc20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 --2- 172.21.15.28:0/3339271910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd7c075730 0x7fcd7c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.914+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 >> 172.21.15.28:0/3339271910 conn(0x7fcd7c07b910 msgr2=0x7fcd7c107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.915+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 shutdown_connections 2024-09-21T05:54:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:54:58.915+0000 7fcd83dae640 1 -- 172.21.15.28:0/3339271910 wait complete. 2024-09-21T05:54:59.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:54:59 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3339271910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:59.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:54:59.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:54:59 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3339271910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:54:59 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3339271910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:54:59.775 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3691612695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 msgr2=0x7fcf14102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3691612695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf14102640 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009900 tx=0x7fcf0402a410 comp rx=0 tx=0).stop 2024-09-21T05:55:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3691612695 shutdown_connections 2024-09-21T05:55:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3691612695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf1410ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3691612695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 0x7fcf14107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.291+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3691612695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf14102640 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.292+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3691612695 >> 172.21.15.28:0/3691612695 conn(0x7fcf14079e40 msgr2=0x7fcf1407a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.292+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3691612695 shutdown_connections 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.292+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3691612695 wait complete. 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.292+0000 7fcf1a88d640 1 Processor -- start 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.292+0000 7fcf1a88d640 1 -- start start 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 0x7fcf1419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf141a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf14112a00 con 0x7fcf14102230 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf14112880 con 0x7fcf141084c0 2024-09-21T05:55:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf1a88d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcf14112b80 con 0x7fcf14107b70 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49886/0 (socket says 172.21.15.28:49886) 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 -- 172.21.15.28:0/3822321931 learned_addr learned my addr 172.21.15.28:0/3822321931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf137fe640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 0x7fcf1419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf141a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 -- 172.21.15.28:0/3822321931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 msgr2=0x7fcf1419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 0x7fcf1419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 -- 172.21.15.28:0/3822321931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 msgr2=0x7fcf141a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf141a1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf13fff640 1 -- 172.21.15.28:0/3822321931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf141a2610 con 0x7fcf14102230 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.293+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf141a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcf04005940 tx=0x7fcf040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf117fa640 1 -- 172.21.15.28:0/3822321931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0403f030 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf117fa640 1 -- 172.21.15.28:0/3822321931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf04003cb0 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf141a28a0 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf117fa640 1 -- 172.21.15.28:0/3822321931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04035070 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.294+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf141a9060 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 Processor -- start 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 -- start start 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf117fa640 1 -- 172.21.15.28:0/3822321931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fcf04042030 con 0x7fcf14102230 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c132e070 0x5641c16d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641c164bea0 con 0x5641c130d390 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5641c163b2d0 con 0x5641c132e070 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf20e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5641c163f0e0 con 0x5641c13051b0 2024-09-21T05:55:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf18e2a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c132e070 0x5641c16d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49894/0 (socket says 172.21.15.28:49894) 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.297+0000 7fcf137fe640 1 -- 172.21.15.28:0/2096825555 learned_addr learned my addr 172.21.15.28:0/2096825555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 -- 172.21.15.28:0/2096825555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 msgr2=0x5641c16d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 -- 172.21.15.28:0/2096825555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c132e070 msgr2=0x5641c16d56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c132e070 0x5641c16d56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 -- 172.21.15.28:0/2096825555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641c1399650 con 0x5641c130d390 2024-09-21T05:55:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf117fa640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 0x7fcefc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf137fe640 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800bc30 tx=0x7fcf080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f3ad905e42e08e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 0x7fcefc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.298+0000 7fcf117fa640 1 -- 172.21.15.28:0/3822321931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fcf04107fd0 con 0x7fcf14102230 2024-09-21T05:55:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.299+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 0x7fcefc0b8b90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005fd0 tx=0x7fcf0c009700 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.300+0000 7fcefa7fc640 1 -- 172.21.15.28:0/2096825555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0801c480 con 0x5641c130d390 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.300+0000 7fcefa7fc640 1 -- 172.21.15.28:0/2096825555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf0801cad0 con 0x5641c130d390 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.300+0000 7fcefa7fc640 1 -- 172.21.15.28:0/2096825555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08011b90 con 0x5641c130d390 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- 172.21.15.28:0/2096825555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 msgr2=0x5641c16d6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800bc30 tx=0x7fcf080107a0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- 172.21.15.28:0/2096825555 shutdown_connections 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c130d390 0x5641c16d6120 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 --2- 172.21.15.28:0/2096825555 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c132e070 0x5641c16d56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- 172.21.15.28:0/2096825555 >> 172.21.15.28:0/2096825555 conn(0x5641c1322460 msgr2=0x5641c16d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- 172.21.15.28:0/2096825555 shutdown_connections 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- 172.21.15.28:0/2096825555 wait complete. 2024-09-21T05:55:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.301+0000 7fcf20e01740 1 -- start start 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 Processor -- start 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c132e070 0x5641c16ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641c149c700 con 0x5641c13051b0 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5641c15d6160 con 0x5641c130d390 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5641c16e3e80 con 0x5641c132e070 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf20e01740 10 client.0 _ll_register_callbacks cb 0x5641c15ef9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49896/0 (socket says 172.21.15.28:49896) 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf13fff640 1 -- 172.21.15.28:0/3781494987 learned_addr learned my addr 172.21.15.28:0/3781494987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf137fe640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.302+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c132e070 0x5641c16ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 -- 172.21.15.28:0/3781494987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c132e070 msgr2=0x5641c16ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c132e070 0x5641c16ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 -- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 msgr2=0x5641c16e1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 -- 172.21.15.28:0/3781494987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641c1603c90 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf13fff640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fcf04003fd0 tx=0x7fcf0403b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195749ccfdff97e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04049070 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fcf040b50b0 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf0801cad0 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcf040cd770 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf20e01740 10 client.17328 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5641c16efbd0 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.303+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf0801cad0 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fcf041059a0 con 0x5641c13051b0 2024-09-21T05:55:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 10 client.17328 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 10 client.17328 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 20 client.17328 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12ec9f0 0x5641c16f3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 4 client.17328 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641c164ddf0 con 0x5641c12ec9f0 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.304+0000 7fcf137fe640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12ec9f0 0x5641c16f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.305+0000 7fcf137fe640 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12ec9f0 0x5641c16f3160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a4bc0d1dd9295ec server_cookie=cc6ffed867cbd646 in_seq=0 out_seq=0 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.305+0000 7fcef37fe640 10 client.17328 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcef37fe640 1 -- 172.21.15.28:0/3781494987 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5641c164ddf0 con 0x5641c12ec9f0 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcef37fe640 10 client.17328 handle_command_reply: tid=0 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 client.17328 shutdown 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 msgr2=0x5641c16e0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fcf04003fd0 tx=0x7fcf0403b6b0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 shutdown_connections 2024-09-21T05:55:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12ec9f0 0x5641c16f3160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c132e070 0x5641c16ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcef37fe640 0 client.17328 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 --2- 172.21.15.28:0/3781494987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c13051b0 0x5641c16e0d40 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 >> 172.21.15.28:0/3781494987 conn(0x5641c1322460 msgr2=0x5641c16d8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 shutdown_connections 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 1 -- 172.21.15.28:0/3781494987 wait complete. 2024-09-21T05:55:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.306+0000 7fcf20e01740 20 client.17328 trim_cache size 0 max 16384 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 Processor -- start 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 -- start start 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141b1920 con 0x5641c132e070 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141b3580 con 0x5641c130d390 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf20e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141b2050 con 0x5641c13051b0 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49912/0 (socket says 172.21.15.28:49912) 2024-09-21T05:55:00.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 -- 172.21.15.28:0/4265322168 learned_addr learned my addr 172.21.15.28:0/4265322168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:00.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf137fe640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf13fff640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 -- 172.21.15.28:0/4265322168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 msgr2=0x5641c16d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 -- 172.21.15.28:0/4265322168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 msgr2=0x5641c16ed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16ed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf18e2a640 1 -- 172.21.15.28:0/4265322168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641c16d3780 con 0x5641c132e070 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.320+0000 7fcf137fe640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16ed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c010740 tx=0x7fcf0c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4fe6b7fe2b4304 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcef9ffb640 1 -- 172.21.15.28:0/4265322168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c010df0 con 0x5641c132e070 2024-09-21T05:55:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcef9ffb640 1 -- 172.21.15.28:0/4265322168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf0c005020 con 0x5641c132e070 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcef9ffb640 1 -- 172.21.15.28:0/4265322168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c027610 con 0x5641c132e070 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcf20e01740 1 -- 172.21.15.28:0/4265322168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 msgr2=0x5641c16ddf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.321+0000 7fcf20e01740 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c010740 tx=0x7fcf0c015000 comp rx=0 tx=0).stop 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 -- 172.21.15.28:0/4265322168 shutdown_connections 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c16ddf30 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16ed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 --2- 172.21.15.28:0/4265322168 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16d6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 -- 172.21.15.28:0/4265322168 >> 172.21.15.28:0/4265322168 conn(0x5641c1322460 msgr2=0x5641c16d66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 -- 172.21.15.28:0/4265322168 shutdown_connections 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 -- 172.21.15.28:0/4265322168 wait complete. 2024-09-21T05:55:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 -- start start 2024-09-21T05:55:00.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 Processor -- start 2024-09-21T05:55:00.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16e5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.322+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf20e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf20e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141b2e80 con 0x5641c132e070 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf20e01740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141b2750 con 0x5641c130d390 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf20e01740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcf141af900 con 0x5641c13051b0 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf20e01740 10 client.0 _ll_register_callbacks cb 0x7fcf04005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49920/0 (socket says 172.21.15.28:49920) 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 -- 172.21.15.28:0/1515262537 learned_addr learned my addr 172.21.15.28:0/1515262537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf13fff640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16e5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf137fe640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 -- 172.21.15.28:0/1515262537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 msgr2=0x5641c16e5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16e5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 -- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 msgr2=0x5641c16e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 -- 172.21.15.28:0/1515262537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641c16e6290 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf137fe640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcf18e2a640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c013fd0 tx=0x7fcf0c004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2516ab74524a94df server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.323+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c035020 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf0c005020 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fcf0c0053e0 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcf0c040050 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 10 client.17334 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5641c175e110 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf0c005020 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fcf0c01e410 con 0x5641c132e070 2024-09-21T05:55:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 10 client.17334 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 10 client.17334 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 20 client.17334 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12c1da0 0x5641c17608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 4 client.17334 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641c16498a0 con 0x5641c12c1da0 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.324+0000 7fcf13fff640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12c1da0 0x5641c17608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcef37fe640 10 client.17334 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf13fff640 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12c1da0 0x5641c17608d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50ac55ed81cab880 server_cookie=64fbb714ff745ae6 in_seq=0 out_seq=0 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcef37fe640 1 -- 172.21.15.28:0/1515262537 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641c16498a0 con 0x5641c12c1da0 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcef37fe640 10 client.17334 handle_command_reply: tid=0 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 client.17334 shutdown 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 msgr2=0x5641c175cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c013fd0 tx=0x7fcf0c004c70 comp rx=0 tx=0).stop 2024-09-21T05:55:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 shutdown_connections 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5641c12c1da0 0x5641c17608d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641c132e070 0x5641c175cb20 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5641c130d390 0x5641c16e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 --2- 172.21.15.28:0/1515262537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5641c13051b0 0x5641c16e5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 >> 172.21.15.28:0/1515262537 conn(0x5641c1322460 msgr2=0x5641c16d64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcef37fe640 0 client.17334 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 shutdown_connections 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.325+0000 7fcf20e01740 1 -- 172.21.15.28:0/1515262537 wait complete. 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf20e01740 20 client.17334 trim_cache size 0 max 16384 2024-09-21T05:55:00.332 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 msgr2=0x7fcefc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 0x7fcefc0b8b90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005fd0 tx=0x7fcf0c009700 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 msgr2=0x7fcf1419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fcf04005940 tx=0x7fcf040042f0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 shutdown_connections 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf141084c0 0x7fcf141a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf14107b70 0x7fcf1419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fcefc0b6680 0x7fcefc0b8b90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 --2- 172.21.15.28:0/3822321931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14102230 0x7fcf1419c6a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 >> 172.21.15.28:0/3822321931 conn(0x7fcf14079e40 msgr2=0x7fcf1410f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 shutdown_connections 2024-09-21T05:55:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:00.326+0000 7fcf1a88d640 1 -- 172.21.15.28:0/3822321931 wait complete. 2024-09-21T05:55:00.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:00 smithi028 ceph-mon[31227]: pgmap v566: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:55:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:00 smithi129 ceph-mon[34885]: pgmap v566: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:55:00.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:00 smithi181 ceph-mon[34819]: pgmap v566: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-09-21T05:55:01.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:55:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 -- 172.21.15.28:0/3626273507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c108470 msgr2=0x7fb66c108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 --2- 172.21.15.28:0/3626273507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c108470 0x7fb66c108880 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb660009900 tx=0x7fb66002a410 comp rx=0 tx=0).stop 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 -- 172.21.15.28:0/3626273507 shutdown_connections 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 --2- 172.21.15.28:0/3626273507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108dc0 0x7fb66c10f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 --2- 172.21.15.28:0/3626273507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c108470 0x7fb66c108880 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 --2- 172.21.15.28:0/3626273507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c103190 0x7fb66c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 -- 172.21.15.28:0/3626273507 >> 172.21.15.28:0/3626273507 conn(0x7fb66c079e40 msgr2=0x7fb66c07a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 -- 172.21.15.28:0/3626273507 shutdown_connections 2024-09-21T05:55:01.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 -- 172.21.15.28:0/3626273507 wait complete. 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.760+0000 7fb6712f7640 1 Processor -- start 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 -- start start 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108470 0x7fb66c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c108dc0 0x7fb66c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c112a80 con 0x7fb66c103190 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c112900 con 0x7fb66c108470 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb6712f7640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c112c00 con 0x7fb66c108dc0 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49936/0 (socket says 172.21.15.28:49936) 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66affd640 1 -- 172.21.15.28:0/2389800202 learned_addr learned my addr 172.21.15.28:0/2389800202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c108dc0 0x7fb66c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108470 0x7fb66c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.761+0000 7fb66affd640 1 -- 172.21.15.28:0/2389800202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c108dc0 msgr2=0x7fb66c19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb66affd640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c108dc0 0x7fb66c19ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb66affd640 1 -- 172.21.15.28:0/2389800202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108470 msgr2=0x7fb66c199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb66affd640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108470 0x7fb66c199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb66affd640 1 -- 172.21.15.28:0/2389800202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c19f340 con 0x7fb66c103190 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb66affd640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009970 tx=0x7fb65c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb653fff640 1 -- 172.21.15.28:0/2389800202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c00d020 con 0x7fb66c103190 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c19f630 con 0x7fb66c103190 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb653fff640 1 -- 172.21.15.28:0/2389800202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65c00d1c0 con 0x7fb66c103190 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb653fff640 1 -- 172.21.15.28:0/2389800202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c00cc50 con 0x7fb66c103190 2024-09-21T05:55:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.762+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c101df0 con 0x7fb66c103190 2024-09-21T05:55:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 Processor -- start 2024-09-21T05:55:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb653fff640 1 -- 172.21.15.28:0/2389800202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb65c007860 con 0x7fb66c103190 2024-09-21T05:55:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 -- start start 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc56cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc4e2ea0 con 0x55bfcc1c5070 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc4d22d0 con 0x55bfcc19c1b0 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb67786b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc4d60e0 con 0x55bfcc1a4390 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49944/0 (socket says 172.21.15.28:49944) 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3604150034 learned_addr learned my addr 172.21.15.28:0/3604150034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.765+0000 7fb66affd640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc56cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3604150034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 msgr2=0x55bfcc56cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc56cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3604150034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 msgr2=0x55bfcc56c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3604150034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfcc230650 con 0x55bfcc1c5070 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc56cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb664007f40 tx=0x7fb66400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ba4d789d7f6645 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb653fff640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 0x7fb6540b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb6517fa640 1 -- 172.21.15.28:0/3604150034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664010900 con 0x55bfcc1c5070 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb66affd640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 0x7fb6540b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb6517fa640 1 -- 172.21.15.28:0/3604150034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6640183f0 con 0x55bfcc1c5070 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.766+0000 7fb6517fa640 1 -- 172.21.15.28:0/3604150034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664017600 con 0x55bfcc1c5070 2024-09-21T05:55:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb653fff640 1 -- 172.21.15.28:0/2389800202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb65c0e8b20 con 0x7fb66c103190 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 -- 172.21.15.28:0/3604150034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 msgr2=0x55bfcc56c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb66affd640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 0x7fb6540b8bc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55bfcc56af40 tx=0x7fb65c0ec610 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb664007f40 tx=0x7fb66400f850 comp rx=0 tx=0).stop 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 -- 172.21.15.28:0/3604150034 shutdown_connections 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc56cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 --2- 172.21.15.28:0/3604150034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc56c2c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 -- 172.21.15.28:0/3604150034 >> 172.21.15.28:0/3604150034 conn(0x55bfcc1b9460 msgr2=0x55bfcc56a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 -- 172.21.15.28:0/3604150034 shutdown_connections 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.767+0000 7fb67786b740 1 -- 172.21.15.28:0/3604150034 wait complete. 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 -- start start 2024-09-21T05:55:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 Processor -- start 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc577f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1c5070 0x55bfcc583dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc333700 con 0x55bfcc19c1b0 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc46d160 con 0x55bfcc1a4390 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55bfcc57aaa0 con 0x55bfcc1c5070 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb67786b740 10 client.0 _ll_register_callbacks cb 0x55bfcc4869e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49958/0 (socket says 172.21.15.28:49958) 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3514469398 learned_addr learned my addr 172.21.15.28:0/3514469398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.768+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc577f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1c5070 0x55bfcc583dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1c5070 msgr2=0x55bfcc583dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1c5070 0x55bfcc583dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3514469398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 msgr2=0x55bfcc577f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc577f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3514469398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfcc49ac90 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc577f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb664005b00 tx=0x7fb664005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8f2c1b4ef5326c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb64eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66402c060 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb664010700 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6640183f0 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb664010900 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb67786b740 10 client.17346 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.769+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55bfcc586840 con 0x55bfcc19c1b0 2024-09-21T05:55:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6640183f0 con 0x55bfcc19c1b0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb664004640 con 0x55bfcc19c1b0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 10 client.17346 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 10 client.17346 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 20 client.17346 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc1839f0 0x55bfcc589da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 4 client.17346 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfcc4e4df0 con 0x55bfcc1839f0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.770+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc1839f0 0x55bfcc589da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.771+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc1839f0 0x55bfcc589da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=22b3b5a6fde79e2b server_cookie=1459d980dc4638d1 in_seq=0 out_seq=0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.771+0000 7fb64eff5640 10 client.17346 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.771+0000 7fb64eff5640 1 -- 172.21.15.28:0/3514469398 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bfcc4e4df0 con 0x55bfcc1839f0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.771+0000 7fb64eff5640 10 client.17346 handle_command_reply: tid=0 2024-09-21T05:55:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 client.17346 shutdown 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 msgr2=0x55bfcc577a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb664005b00 tx=0x7fb664005b70 comp rx=0 tx=0).stop 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 shutdown_connections 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc1839f0 0x55bfcc589da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1c5070 0x55bfcc583dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc577f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 --2- 172.21.15.28:0/3514469398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc19c1b0 0x55bfcc577a20 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 >> 172.21.15.28:0/3514469398 conn(0x55bfcc1b9460 msgr2=0x55bfcc56f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb64eff5640 0 client.17346 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 shutdown_connections 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 1 -- 172.21.15.28:0/3514469398 wait complete. 2024-09-21T05:55:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.772+0000 7fb67786b740 20 client.17346 trim_cache size 0 max 16384 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 Processor -- start 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 -- start start 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc583d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.785+0000 7fb67786b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1ad600 con 0x55bfcc1c5070 2024-09-21T05:55:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb67786b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1af260 con 0x55bfcc19c1b0 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb67786b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1add30 con 0x55bfcc1a4390 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49960/0 (socket says 172.21.15.28:49960) 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 -- 172.21.15.28:0/663924249 learned_addr learned my addr 172.21.15.28:0/663924249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 -- 172.21.15.28:0/663924249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 msgr2=0x55bfcc583d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc583d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc583d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 -- 172.21.15.28:0/663924249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 msgr2=0x55bfcc56fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 -- 172.21.15.28:0/663924249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfcc57a5d0 con 0x55bfcc1c5070 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x55bfcc4707a0 tx=0x7fb664005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228b34c1383ca985 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb650ff9640 1 -- 172.21.15.28:0/663924249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66402c070 con 0x55bfcc1c5070 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb650ff9640 1 -- 172.21.15.28:0/663924249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6640183f0 con 0x55bfcc1c5070 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.786+0000 7fb650ff9640 1 -- 172.21.15.28:0/663924249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664010aa0 con 0x55bfcc1c5070 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 -- 172.21.15.28:0/663924249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 msgr2=0x55bfcc574c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x55bfcc4707a0 tx=0x7fb664005040 comp rx=0 tx=0).stop 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 -- 172.21.15.28:0/663924249 shutdown_connections 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc574c10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc1a4390 0x55bfcc583d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 --2- 172.21.15.28:0/663924249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc19c1b0 0x55bfcc56fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 -- 172.21.15.28:0/663924249 >> 172.21.15.28:0/663924249 conn(0x55bfcc1b9460 msgr2=0x55bfcc56f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 -- 172.21.15.28:0/663924249 shutdown_connections 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.787+0000 7fb67786b740 1 -- 172.21.15.28:0/663924249 wait complete. 2024-09-21T05:55:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 -- start start 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 Processor -- start 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc19c1b0 0x55bfcc58a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc58aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1aeb60 con 0x55bfcc1c5070 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1ae430 con 0x55bfcc1a4390 2024-09-21T05:55:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1ab5e0 con 0x55bfcc19c1b0 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb67786b740 10 client.0 _ll_register_callbacks cb 0x7fb65c0ec4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49962/0 (socket says 172.21.15.28:49962) 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3130842010 learned_addr learned my addr 172.21.15.28:0/3130842010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc19c1b0 0x55bfcc58a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.788+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc58aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc19c1b0 msgr2=0x55bfcc58a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc19c1b0 0x55bfcc58a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3130842010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 msgr2=0x55bfcc58aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc58aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 -- 172.21.15.28:0/3130842010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfcc5ecd40 con 0x55bfcc1c5070 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc58aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb664027780 tx=0x7fb664005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cab8abd519699d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb64eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66402c040 con 0x55bfcc1c5070 2024-09-21T05:55:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fb664002030 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6640183f0 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb664025b00 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb67786b740 10 client.17352 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55bfcc5f4130 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.789+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6640183f0 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fb664025400 con 0x55bfcc1c5070 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 10 client.17352 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 10 client.17352 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 20 client.17352 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc158da0 0x55bfcc5f6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 4 client.17352 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfcc4e08a0 con 0x55bfcc158da0 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc158da0 0x55bfcc5f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb66a7fc640 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc158da0 0x55bfcc5f6740 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1db26f6f649aa4d8 server_cookie=f93f77cf86aba248 in_seq=0 out_seq=0 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.790+0000 7fb64eff5640 10 client.17352 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb64eff5640 1 -- 172.21.15.28:0/3130842010 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bfcc4e08a0 con 0x55bfcc158da0 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb64eff5640 10 client.17352 handle_command_reply: tid=0 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 client.17352 shutdown 2024-09-21T05:55:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 msgr2=0x55bfcc5ec6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb664027780 tx=0x7fb664005b70 comp rx=0 tx=0).stop 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 shutdown_connections 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55bfcc158da0 0x55bfcc5f6740 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfcc1c5070 0x55bfcc5ec6d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfcc1a4390 0x55bfcc58aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 --2- 172.21.15.28:0/3130842010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bfcc19c1b0 0x55bfcc58a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 >> 172.21.15.28:0/3130842010 conn(0x55bfcc1b9460 msgr2=0x55bfcc578ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb64eff5640 0 client.17352 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 shutdown_connections 2024-09-21T05:55:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 1 -- 172.21.15.28:0/3130842010 wait complete. 2024-09-21T05:55:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.791+0000 7fb67786b740 20 client.17352 trim_cache size 0 max 16384 2024-09-21T05:55:01.804 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 msgr2=0x7fb6540b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 0x7fb6540b8bc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55bfcc56af40 tx=0x7fb65c0ec610 comp rx=0 tx=0).stop 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 msgr2=0x7fb66c199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009970 tx=0x7fb65c009e30 comp rx=0 tx=0).stop 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 shutdown_connections 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb66c108dc0 0x7fb66c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb66c108470 0x7fb66c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb6540b66b0 0x7fb6540b8bc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 --2- 172.21.15.28:0/2389800202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c103190 0x7fb66c199490 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 >> 172.21.15.28:0/2389800202 conn(0x7fb66c079e40 msgr2=0x7fb66c104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 shutdown_connections 2024-09-21T05:55:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:01.792+0000 7fb6712f7640 1 -- 172.21.15.28:0/2389800202 wait complete. 2024-09-21T05:55:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:02 smithi028 ceph-mon[31227]: pgmap v567: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:02 smithi129 ceph-mon[34885]: pgmap v567: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:02.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:55:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:02 smithi181 ceph-mon[34819]: pgmap v567: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:02.955 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 -- 172.21.15.28:0/2686802726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa24010deb0 msgr2=0x7fa240110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 --2- 172.21.15.28:0/2686802726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa24010deb0 0x7fa240110330 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa238009ca0 tx=0x7fa23802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 -- 172.21.15.28:0/2686802726 shutdown_connections 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 --2- 172.21.15.28:0/2686802726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa24010deb0 0x7fa240110330 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 --2- 172.21.15.28:0/2686802726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 --2- 172.21.15.28:0/2686802726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa240104900 0x7fa240104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 -- 172.21.15.28:0/2686802726 >> 172.21.15.28:0/2686802726 conn(0x7fa240079760 msgr2=0x7fa240079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 -- 172.21.15.28:0/2686802726 shutdown_connections 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.476+0000 7fa246522640 1 -- 172.21.15.28:0/2686802726 wait complete. 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.477+0000 7fa246522640 1 Processor -- start 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.477+0000 7fa246522640 1 -- start start 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa24010deb0 0x7fa24019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa240113aa0 con 0x7fa240104900 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa240113920 con 0x7fa240105250 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa246522640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa240113c20 con 0x7fa24010deb0 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49980/0 (socket says 172.21.15.28:49980) 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 -- 172.21.15.28:0/4171514776 learned_addr learned my addr 172.21.15.28:0/4171514776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa244abf640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa24010deb0 0x7fa24019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 -- 172.21.15.28:0/4171514776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa24010deb0 msgr2=0x7fa24019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa24010deb0 0x7fa24019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 -- 172.21.15.28:0/4171514776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 msgr2=0x7fa24019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23ffff640 1 -- 172.21.15.28:0/4171514776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2401a02a0 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.478+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa23ffff640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa23000d770 tx=0x7fa23000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa23d7fa640 1 -- 172.21.15.28:0/4171514776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230017070 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2401a0590 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa23d7fa640 1 -- 172.21.15.28:0/4171514776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa230009c30 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa23d7fa640 1 -- 172.21.15.28:0/4171514776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230003cc0 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.479+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401a6db0 con 0x7fa240104900 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.481+0000 7fa24ca96740 1 Processor -- start 2024-09-21T05:55:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 -- start start 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa23d7fa640 1 -- 172.21.15.28:0/4171514776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa230003e60 con 0x7fa240104900 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc05be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc06120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563ebb7bea0 con 0x5563eb85e070 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563ebb6b2d0 con 0x5563eb8351b0 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa24ca96740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5563ebb6f0e0 con 0x5563eb83d390 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49992/0 (socket says 172.21.15.28:49992) 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 -- 172.21.15.28:0/1060834801 learned_addr learned my addr 172.21.15.28:0/1060834801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc06120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa23ffff640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc05be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 -- 172.21.15.28:0/1060834801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 msgr2=0x5563ebc06120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc06120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 -- 172.21.15.28:0/1060834801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 msgr2=0x5563ebc05be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc05be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa244abf640 1 -- 172.21.15.28:0/1060834801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563eb8c9650 con 0x5563eb85e070 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.482+0000 7fa23ffff640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc05be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa244abf640 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fa238035000 tx=0x7fa238009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73205d585721036b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa23d7fa640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 0x7fa2280b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa2267fc640 1 -- 172.21.15.28:0/1060834801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238031df0 con 0x5563eb85e070 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa23ffff640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 0x7fa2280b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa2267fc640 1 -- 172.21.15.28:0/1060834801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa238036040 con 0x5563eb85e070 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa2267fc640 1 -- 172.21.15.28:0/1060834801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238004330 con 0x5563eb85e070 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.483+0000 7fa23d7fa640 1 -- 172.21.15.28:0/4171514776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fa2300e7150 con 0x7fa240104900 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- 172.21.15.28:0/1060834801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 msgr2=0x5563ebc056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fa238035000 tx=0x7fa238009930 comp rx=0 tx=0).stop 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa23ffff640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 0x7fa2280b8bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5563ebc04320 tx=0x7fa230094040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- 172.21.15.28:0/1060834801 shutdown_connections 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc06120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc05be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1060834801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc056a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- 172.21.15.28:0/1060834801 >> 172.21.15.28:0/1060834801 conn(0x5563eb852460 msgr2=0x5563ebc03590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- 172.21.15.28:0/1060834801 shutdown_connections 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- 172.21.15.28:0/1060834801 wait complete. 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.484+0000 7fa24ca96740 1 -- start start 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 Processor -- start 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc11320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb85e070 0x5563ebc1d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563eb9cc700 con 0x5563eb8351b0 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563ebb06160 con 0x5563eb85e070 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5563ebc13e20 con 0x5563eb83d390 2024-09-21T05:55:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa24ca96740 10 client.0 _ll_register_callbacks cb 0x5563ebb1f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50008/0 (socket says 172.21.15.28:50008) 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 -- 172.21.15.28:0/105775890 learned_addr learned my addr 172.21.15.28:0/105775890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc11320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 -- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 msgr2=0x5563ebc11320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc11320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 -- 172.21.15.28:0/105775890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb85e070 msgr2=0x5563ebc1d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb85e070 0x5563ebc1d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.485+0000 7fa244abf640 1 -- 172.21.15.28:0/105775890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ebb33c90 con 0x5563eb8351b0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa244abf640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa238006010 tx=0x7fa2380095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e2f5a30f502dc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238031510 con 0x5563eb8351b0 2024-09-21T05:55:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa238036040 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fa2380316b0 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa23805d050 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa24ca96740 10 client.17364 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5563ebc1fb00 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.486+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa238036040 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fa238058030 con 0x5563eb8351b0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 10 client.17364 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 10 client.17364 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 20 client.17364 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb81c9f0 0x5563ebc22e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 4 client.17364 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563ebb7ddf0 con 0x5563eb81c9f0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb81c9f0 0x5563ebc22e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa21f7fe640 10 client.17364 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.487+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb81c9f0 0x5563ebc22e60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a757d747d2908a02 server_cookie=2ee3e1b7aa60d8f0 in_seq=0 out_seq=0 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa21f7fe640 1 -- 172.21.15.28:0/105775890 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5563ebb7ddf0 con 0x5563eb81c9f0 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa21f7fe640 10 client.17364 handle_command_reply: tid=0 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 client.17364 shutdown 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 msgr2=0x5563ebc10de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa238006010 tx=0x7fa2380095e0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 shutdown_connections 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb81c9f0 0x5563ebc22e60 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb85e070 0x5563ebc1d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc11320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 --2- 172.21.15.28:0/105775890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb8351b0 0x5563ebc10de0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa21f7fe640 0 client.17364 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.488+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 >> 172.21.15.28:0/105775890 conn(0x5563eb852460 msgr2=0x5563ebc08850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.489+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 shutdown_connections 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.489+0000 7fa24ca96740 1 -- 172.21.15.28:0/105775890 wait complete. 2024-09-21T05:55:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.489+0000 7fa24ca96740 20 client.17364 trim_cache size 0 max 16384 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.502+0000 7fa24ca96740 1 Processor -- start 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.502+0000 7fa24ca96740 1 -- start start 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc08eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc0dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401b18b0 con 0x5563eb85e070 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401b3510 con 0x5563eb8351b0 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa24ca96740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401b1fe0 con 0x5563eb83d390 2024-09-21T05:55:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50018/0 (socket says 172.21.15.28:50018) 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 -- 172.21.15.28:0/1429987432 learned_addr learned my addr 172.21.15.28:0/1429987432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc08eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc0dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 -- 172.21.15.28:0/1429987432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 msgr2=0x5563ebc0dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc0dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 -- 172.21.15.28:0/1429987432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 msgr2=0x5563ebc08eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc08eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 -- 172.21.15.28:0/1429987432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ebc07be0 con 0x5563eb85e070 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.503+0000 7fa244abf640 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa238005d50 tx=0x7fa238009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39db674d7f49438c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa225ffb640 1 -- 172.21.15.28:0/1429987432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23802f660 con 0x5563eb85e070 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa225ffb640 1 -- 172.21.15.28:0/1429987432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa238036040 con 0x5563eb85e070 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa225ffb640 1 -- 172.21.15.28:0/1429987432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238003b80 con 0x5563eb85e070 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa24ca96740 1 -- 172.21.15.28:0/1429987432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 msgr2=0x5563ebc0e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa238005d50 tx=0x7fa238009540 comp rx=0 tx=0).stop 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa24ca96740 1 -- 172.21.15.28:0/1429987432 shutdown_connections 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc0e510 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.504+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc0dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 --2- 172.21.15.28:0/1429987432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc08eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 -- 172.21.15.28:0/1429987432 >> 172.21.15.28:0/1429987432 conn(0x5563eb852460 msgr2=0x5563ebc087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 -- 172.21.15.28:0/1429987432 shutdown_connections 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 -- 172.21.15.28:0/1429987432 wait complete. 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 -- start start 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.505+0000 7fa24ca96740 1 Processor -- start 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc24080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc245c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401b2e10 con 0x5563eb85e070 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401b26e0 con 0x5563eb8351b0 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401af890 con 0x5563eb83d390 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa24ca96740 10 client.0 _ll_register_callbacks cb 0x7fa238009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc245c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50024/0 (socket says 172.21.15.28:50024) 2024-09-21T05:55:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 -- 172.21.15.28:0/2624269837 learned_addr learned my addr 172.21.15.28:0/2624269837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc24080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 -- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 msgr2=0x5563ebc245c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc245c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 -- 172.21.15.28:0/2624269837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 msgr2=0x5563ebc24080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc24080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa244abf640 1 -- 172.21.15.28:0/2624269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ebc8c7e0 con 0x5563eb85e070 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.506+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc245c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa244abf640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x5563ebc10950 tx=0x7fa23804a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8474604fdb7fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238068040 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7fa238031510 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa238036040 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa238031710 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 10 client.17370 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5563ebc8da60 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa238036040 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7fa238058040 con 0x5563eb85e070 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 10 client.17370 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 10 client.17370 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 20 client.17370 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb7f1da0 0x5563ebc901e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 4 client.17370 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.507+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563ebb798a0 con 0x5563eb7f1da0 2024-09-21T05:55:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb7f1da0 0x5563ebc901e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa21f7fe640 10 client.17370 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa23f7fe640 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb7f1da0 0x5563ebc901e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7826d3d419a07ac5 server_cookie=923fdcd0f478f49 in_seq=0 out_seq=0 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa21f7fe640 1 -- 172.21.15.28:0/2624269837 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563ebb798a0 con 0x5563eb7f1da0 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa21f7fe640 10 client.17370 handle_command_reply: tid=0 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.508+0000 7fa24ca96740 1 client.17370 shutdown 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 msgr2=0x5563ebc24b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x5563ebc10950 tx=0x7fa23804a040 comp rx=0 tx=0).stop 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 shutdown_connections 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x5563eb7f1da0 0x5563ebc901e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563eb85e070 0x5563ebc24b00 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5563eb83d390 0x5563ebc245c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 --2- 172.21.15.28:0/2624269837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563eb8351b0 0x5563ebc24080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 >> 172.21.15.28:0/2624269837 conn(0x5563eb852460 msgr2=0x5563ebc065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa21f7fe640 0 client.17370 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 shutdown_connections 2024-09-21T05:55:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 1 -- 172.21.15.28:0/2624269837 wait complete. 2024-09-21T05:55:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa24ca96740 20 client.17370 trim_cache size 0 max 16384 2024-09-21T05:55:03.515 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 msgr2=0x7fa2280b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 0x7fa2280b8bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5563ebc04320 tx=0x7fa230094040 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 msgr2=0x7fa24019a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.509+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa23000d770 tx=0x7fa23000dc30 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 shutdown_connections 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa24010deb0 0x7fa24019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa240105250 0x7fa24019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa2280b66b0 0x7fa2280b8bc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 --2- 172.21.15.28:0/4171514776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa240104900 0x7fa24019a3f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 >> 172.21.15.28:0/4171514776 conn(0x7fa240079760 msgr2=0x7fa240105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 shutdown_connections 2024-09-21T05:55:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:03.510+0000 7fa246522640 1 -- 172.21.15.28:0/4171514776 wait complete. 2024-09-21T05:55:04.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:55:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:04.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:04 smithi028 ceph-mon[31227]: pgmap v568: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:55:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:04 smithi129 ceph-mon[34885]: pgmap v568: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:55:04.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:04 smithi181 ceph-mon[34819]: pgmap v568: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 -- 172.21.15.28:0/1008667878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 msgr2=0x7f71d01087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 --2- 172.21.15.28:0/1008667878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d01087b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009900 tx=0x7f71c402a410 comp rx=0 tx=0).stop 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 -- 172.21.15.28:0/1008667878 shutdown_connections 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 --2- 172.21.15.28:0/1008667878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0108cf0 0x7f71d010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 --2- 172.21.15.28:0/1008667878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d01087b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 --2- 172.21.15.28:0/1008667878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0102a60 0x7f71d0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.940+0000 7f71d62a6640 1 -- 172.21.15.28:0/1008667878 >> 172.21.15.28:0/1008667878 conn(0x7f71d0079730 msgr2=0x7f71d0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.941+0000 7f71d62a6640 1 -- 172.21.15.28:0/1008667878 shutdown_connections 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.941+0000 7f71d62a6640 1 -- 172.21.15.28:0/1008667878 wait complete. 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.941+0000 7f71d62a6640 1 Processor -- start 2024-09-21T05:55:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 -- start start 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0102a60 0x7f71d019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 0x7f71d019be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0112960 con 0x7f71d01083a0 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01127e0 con 0x7f71d0108cf0 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d62a6640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0112ae0 con 0x7f71d0102a60 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50038/0 (socket says 172.21.15.28:50038) 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71cf7fe640 1 -- 172.21.15.28:0/260063021 learned_addr learned my addr 172.21.15.28:0/260063021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71cffff640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0102a60 0x7f71d019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.942+0000 7f71d4843640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 0x7f71d019be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 -- 172.21.15.28:0/260063021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0102a60 msgr2=0x7f71d019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0102a60 0x7f71d019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 -- 172.21.15.28:0/260063021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 msgr2=0x7f71d019be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 0x7f71d019be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 -- 172.21.15.28:0/260063021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d01a6620 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71d4843640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 0x7f71d019be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f71c4032fd0 tx=0x7f71c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cd7fa640 1 -- 172.21.15.28:0/260063021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c403f030 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cd7fa640 1 -- 172.21.15.28:0/260063021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71c4004490 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d01a68b0 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71cd7fa640 1 -- 172.21.15.28:0/260063021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4035070 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.943+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d01a6d90 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 Processor -- start 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 -- start start 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71cd7fa640 1 -- 172.21.15.28:0/260063021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f71c4042030 con 0x7f71d01083a0 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e876070 0x55ce8ec1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce8eb93ea0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce8eb832d0 con 0x55ce8e876070 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71dc81a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ce8eb870e0 con 0x55ce8e855390 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50044/0 (socket says 172.21.15.28:50044) 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71cffff640 1 -- 172.21.15.28:0/3975271847 learned_addr learned my addr 172.21.15.28:0/3975271847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71d4843640 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e876070 0x55ce8ec1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.946+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 -- 172.21.15.28:0/3975271847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 msgr2=0x55ce8ec1dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec1dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 -- 172.21.15.28:0/3975271847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e876070 msgr2=0x55ce8ec1d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e876070 0x55ce8ec1d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 -- 172.21.15.28:0/3975271847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce8e8e1650 con 0x55ce8e84d1b0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cffff640 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f71c000bcb0 tx=0x7f71c00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3b19f0475ca42a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cd7fa640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 0x7f71b80b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71d4843640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 0x7f71b80b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71b67fc640 1 -- 172.21.15.28:0/3975271847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c001c480 con 0x55ce8e84d1b0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71cd7fa640 1 -- 172.21.15.28:0/260063021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f71c4107c50 con 0x7f71d01083a0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71b67fc640 1 -- 172.21.15.28:0/3975271847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71c001cad0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.947+0000 7f71b67fc640 1 -- 172.21.15.28:0/3975271847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0011ce0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 -- 172.21.15.28:0/3975271847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 msgr2=0x55ce8ec1d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71d4843640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 0x7f71b80b8bc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f71c8005fd0 tx=0x7f71c8009780 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f71c000bcb0 tx=0x7f71c00107a0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 -- 172.21.15.28:0/3975271847 shutdown_connections 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec1dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec1d800 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3975271847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e876070 0x55ce8ec1d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 -- 172.21.15.28:0/3975271847 >> 172.21.15.28:0/3975271847 conn(0x55ce8e86a460 msgr2=0x55ce8ec1b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 -- 172.21.15.28:0/3975271847 shutdown_connections 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.948+0000 7f71dc81a740 1 -- 172.21.15.28:0/3975271847 wait complete. 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 -- start start 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 Processor -- start 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec28ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 0x55ce8ec34dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce8e9e4700 con 0x55ce8e84d1b0 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce8eb1e160 con 0x55ce8e855390 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ce8ec2baa0 con 0x55ce8e876070 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71dc81a740 10 client.0 _ll_register_callbacks cb 0x55ce8eb379e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50056/0 (socket says 172.21.15.28:50056) 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.949+0000 7f71cffff640 1 -- 172.21.15.28:0/3121627954 learned_addr learned my addr 172.21.15.28:0/3121627954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec28ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71d4843640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 0x55ce8ec34dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 -- 172.21.15.28:0/3121627954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 msgr2=0x55ce8ec34dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 0x55ce8ec34dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 -- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 msgr2=0x55ce8ec28ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec28ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 -- 172.21.15.28:0/3121627954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce8eb4bc90 con 0x55ce8e84d1b0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71d4843640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 0x55ce8ec34dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71cffff640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f71c0011010 tx=0x7f71c000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a15bc97c9efa63 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c002b060 con 0x55ce8e84d1b0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f71c0011be0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71c001cad0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f71c0011de0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71dc81a740 10 client.17382 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.950+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ce8ec37780 con 0x55ce8e84d1b0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71c001cad0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f71c0025070 con 0x55ce8e84d1b0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 10 client.17382 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 10 client.17382 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 20 client.17382 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e8349f0 0x55ce8ec3ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 4 client.17382 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce8eb95df0 con 0x55ce8e8349f0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.951+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e8349f0 0x55ce8ec3ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.952+0000 7f71af7fe640 10 client.17382 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.952+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e8349f0 0x55ce8ec3ab90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b672571c3fa38c13 server_cookie=a8b6332e00e1236 in_seq=0 out_seq=0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71af7fe640 1 -- 172.21.15.28:0/3121627954 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ce8eb95df0 con 0x55ce8e8349f0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71af7fe640 10 client.17382 handle_command_reply: tid=0 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 client.17382 shutdown 2024-09-21T05:55:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 msgr2=0x55ce8ec28980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f71c0011010 tx=0x7f71c000bfd0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 shutdown_connections 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e8349f0 0x55ce8ec3ab90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e876070 0x55ce8ec34dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71af7fe640 0 client.17382 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec28ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3121627954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec28980 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 >> 172.21.15.28:0/3121627954 conn(0x55ce8e86a460 msgr2=0x55ce8ec20450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 shutdown_connections 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 1 -- 172.21.15.28:0/3121627954 wait complete. 2024-09-21T05:55:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.953+0000 7f71dc81a740 20 client.17382 trim_cache size 0 max 16384 2024-09-21T05:55:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 Processor -- start 2024-09-21T05:55:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 -- start start 2024-09-21T05:55:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e8349f0 0x55ce8ec28e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec25b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01af7c0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b1420 con 0x55ce8e855390 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71dc81a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01afef0 con 0x55ce8e8349f0 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50058/0 (socket says 172.21.15.28:50058) 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3529462012 learned_addr learned my addr 172.21.15.28:0/3529462012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71d4843640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec25b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.967+0000 7f71cffff640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e8349f0 0x55ce8ec28e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3529462012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e8349f0 msgr2=0x55ce8ec28e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e8349f0 0x55ce8ec28e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3529462012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 msgr2=0x55ce8ec25b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec25b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 -- 172.21.15.28:0/3529462012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce8ec165d0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71d4843640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec25b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f71c4031440 tx=0x7f71c403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683b86a4171c8a91 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71b5ffb640 1 -- 172.21.15.28:0/3529462012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4049040 con 0x55ce8e84d1b0 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71b5ffb640 1 -- 172.21.15.28:0/3529462012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71c40cd5b0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.968+0000 7f71b5ffb640 1 -- 172.21.15.28:0/3529462012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40c9d10 con 0x55ce8e84d1b0 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- 172.21.15.28:0/3529462012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 msgr2=0x55ce8ec34d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f71c4031440 tx=0x7f71c403b000 comp rx=0 tx=0).stop 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- 172.21.15.28:0/3529462012 shutdown_connections 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e855390 0x55ce8ec25b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec34d30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3529462012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e8349f0 0x55ce8ec28e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- 172.21.15.28:0/3529462012 >> 172.21.15.28:0/3529462012 conn(0x55ce8e86a460 msgr2=0x55ce8ec28730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- 172.21.15.28:0/3529462012 shutdown_connections 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- 172.21.15.28:0/3529462012 wait complete. 2024-09-21T05:55:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.969+0000 7f71dc81a740 1 -- start start 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 Processor -- start 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec3bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b0d20 con 0x55ce8e8349f0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b05f0 con 0x55ce8e84d1b0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01ad7a0 con 0x55ce8e855390 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71dc81a740 10 client.0 _ll_register_callbacks cb 0x7f71c8009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50070/0 (socket says 172.21.15.28:50070) 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71cffff640 1 -- 172.21.15.28:0/3045798388 learned_addr learned my addr 172.21.15.28:0/3045798388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71d4843640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec3bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.970+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 -- 172.21.15.28:0/3045798388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 msgr2=0x55ce8ec3bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec3bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 -- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e84d1b0 msgr2=0x55ce8ec3b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec3b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 -- 172.21.15.28:0/3045798388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce8ec3c290 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71cffff640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f71c0000c00 tx=0x7f71c000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684cbea56bbf7fa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c002b040 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(75..75 src has 1..75) ==== 8528+0+0 (secure 0 0 0) 0x7f71c0004040 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71c40cd5b0 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f71c0004280 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71dc81a740 10 client.17388 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.971+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ce8eca50e0 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71c40cd5b0 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f71c0025040 con 0x55ce8e8349f0 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 10 client.17388 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 10 client.17388 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 20 client.17388 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e876070 0x55ce8eca7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 4 client.17388 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce8eb918a0 con 0x55ce8e876070 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.972+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e876070 0x55ce8eca7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71af7fe640 10 client.17388 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71cf7fe640 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e876070 0x55ce8eca7920 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98b4b7f61e62b8f0 server_cookie=d895dc8c4b3114c4 in_seq=0 out_seq=0 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71af7fe640 1 -- 172.21.15.28:0/3045798388 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce8eb918a0 con 0x55ce8e876070 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71af7fe640 10 client.17388 handle_command_reply: tid=0 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 client.17388 shutdown 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 msgr2=0x55ce8ec3e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f71c0000c00 tx=0x7f71c000b5e0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 shutdown_connections 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55ce8e876070 0x55ce8eca7920 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ce8e855390 0x55ce8ec3bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce8e84d1b0 0x55ce8ec3b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 --2- 172.21.15.28:0/3045798388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8e8349f0 0x55ce8ec3e240 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 >> 172.21.15.28:0/3045798388 conn(0x55ce8e86a460 msgr2=0x55ce8ec1e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71af7fe640 0 client.17388 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 shutdown_connections 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 1 -- 172.21.15.28:0/3045798388 wait complete. 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.973+0000 7f71dc81a740 20 client.17388 trim_cache size 0 max 16384 2024-09-21T05:55:04.986 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 msgr2=0x7f71b80b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 0x7f71b80b8bc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f71c8005fd0 tx=0x7f71c8009780 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 msgr2=0x7f71d019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f71c4032fd0 tx=0x7f71c4004270 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 shutdown_connections 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d0108cf0 0x7f71d019be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f71b80b66b0 0x7f71b80b8bc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d01083a0 0x7f71d019b8d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 --2- 172.21.15.28:0/260063021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d0102a60 0x7f71d019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 >> 172.21.15.28:0/260063021 conn(0x7f71d0079730 msgr2=0x7f71d0104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 shutdown_connections 2024-09-21T05:55:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:04.974+0000 7f71d62a6640 1 -- 172.21.15.28:0/260063021 wait complete. 2024-09-21T05:55:05.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:55:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:06.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.418+0000 7ff654c96640 1 -- 172.21.15.28:0/3724760716 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c037070 con 0x7ff650108cf0 2024-09-21T05:55:06.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.419+0000 7ff657efc640 1 -- 172.21.15.28:0/3724760716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650108cf0 msgr2=0x7ff65010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.419+0000 7ff657efc640 1 --2- 172.21.15.28:0/3724760716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650108cf0 0x7ff65010f520 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009860 tx=0x7ff64c02a360 comp rx=0 tx=0).stop 2024-09-21T05:55:06.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.420+0000 7ff657efc640 1 -- 172.21.15.28:0/3724760716 shutdown_connections 2024-09-21T05:55:06.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.420+0000 7ff657efc640 1 --2- 172.21.15.28:0/3724760716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650108cf0 0x7ff65010f520 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.420+0000 7ff657efc640 1 --2- 172.21.15.28:0/3724760716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6501083a0 0x7ff6501087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.420+0000 7ff657efc640 1 --2- 172.21.15.28:0/3724760716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650102a60 0x7ff650102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.420+0000 7ff657efc640 1 -- 172.21.15.28:0/3724760716 >> 172.21.15.28:0/3724760716 conn(0x7ff650079730 msgr2=0x7ff650079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.422+0000 7ff657efc640 1 -- 172.21.15.28:0/3724760716 shutdown_connections 2024-09-21T05:55:06.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.422+0000 7ff657efc640 1 -- 172.21.15.28:0/3724760716 wait complete. 2024-09-21T05:55:06.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 Processor -- start 2024-09-21T05:55:06.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 -- start start 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6501083a0 0x7ff65010a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 0x7ff65010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff65006f6d0 con 0x7ff650102a60 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff65006f550 con 0x7ff6501083a0 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.423+0000 7ff657efc640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff65006f850 con 0x7ff650108cf0 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff655c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff655c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46736/0 (socket says 172.21.15.28:46736) 2024-09-21T05:55:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff655c98640 1 -- 172.21.15.28:0/3389809007 learned_addr learned my addr 172.21.15.28:0/3389809007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 0x7ff65010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 -- 172.21.15.28:0/3389809007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6501083a0 msgr2=0x7ff65010a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6501083a0 0x7ff65010a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 -- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 msgr2=0x7ff65010f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.424+0000 7ff656499640 1 -- 172.21.15.28:0/3389809007 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65010b1d0 con 0x7ff650108cf0 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.425+0000 7ff655c98640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.425+0000 7ff656499640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 0x7ff65010aa50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff64c035040 tx=0x7ff64c033b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.426+0000 7ff646ffd640 1 -- 172.21.15.28:0/3389809007 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c03f020 con 0x7ff650108cf0 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.426+0000 7ff646ffd640 1 -- 172.21.15.28:0/3389809007 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff64c005050 con 0x7ff650108cf0 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.426+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6501c6690 con 0x7ff650108cf0 2024-09-21T05:55:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.426+0000 7ff646ffd640 1 -- 172.21.15.28:0/3389809007 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c037040 con 0x7ff650108cf0 2024-09-21T05:55:06.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.426+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6501c6ae0 con 0x7ff650108cf0 2024-09-21T05:55:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.432+0000 7ff65e470740 1 Processor -- start 2024-09-21T05:55:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.434+0000 7ff65e470740 1 -- start start 2024-09-21T05:55:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.434+0000 7ff646ffd640 1 -- 172.21.15.28:0/3389809007 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff64c042020 con 0x7ff650108cf0 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.434+0000 7ff65e470740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fe5070 0x55d39338c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff65e470740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d39338c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff65e470740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff65e470740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d393302ea0 con 0x55d392fe5070 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff65e470740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d3932f22d0 con 0x55d392fc4390 2024-09-21T05:55:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff65e470740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d3932f60e0 con 0x55d392fbc1b0 2024-09-21T05:55:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff655497640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff655497640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:50580/0 (socket says 172.21.15.28:50580) 2024-09-21T05:55:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff655497640 1 -- 172.21.15.28:0/1894443508 learned_addr learned my addr 172.21.15.28:0/1894443508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff655497640 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fe5070 0x55d39338c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.435+0000 7ff655c98640 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d39338c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 -- 172.21.15.28:0/1894443508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 msgr2=0x55d39338c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d39338c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 -- 172.21.15.28:0/1894443508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fe5070 msgr2=0x55d39338c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fe5070 0x55d39338c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 -- 172.21.15.28:0/1894443508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d393050650 con 0x55d392fc4390 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655497640 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff6400121b0 tx=0x7ff64000b4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b64273f2b09349b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff646ffd640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 0x7ff63c0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff655c98640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 0x7ff63c0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.436+0000 7ff646ffd640 1 -- 172.21.15.28:0/3389809007 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7ff64c106ea0 con 0x7ff650108cf0 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.437+0000 7ff63bfff640 1 -- 172.21.15.28:0/1894443508 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640004870 con 0x55d392fc4390 2024-09-21T05:55:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.437+0000 7ff63bfff640 1 -- 172.21.15.28:0/1894443508 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6400060d0 con 0x55d392fc4390 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.438+0000 7ff65e470740 1 -- 172.21.15.28:0/1894443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 msgr2=0x55d39338cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.438+0000 7ff65e470740 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff6400121b0 tx=0x7ff64000b4d0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.438+0000 7ff655c98640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 0x7ff63c0b8d60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55d39338af40 tx=0x7ff64801a5d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.439+0000 7ff65e470740 1 -- 172.21.15.28:0/1894443508 shutdown_connections 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.439+0000 7ff65e470740 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39338cd40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.439+0000 7ff65e470740 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d39338c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.439+0000 7ff65e470740 1 --2- 172.21.15.28:0/1894443508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fe5070 0x55d39338c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.439+0000 7ff65e470740 1 -- 172.21.15.28:0/1894443508 >> 172.21.15.28:0/1894443508 conn(0x55d392fd9460 msgr2=0x55d39338a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.440+0000 7ff65e470740 1 -- 172.21.15.28:0/1894443508 shutdown_connections 2024-09-21T05:55:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.440+0000 7ff65e470740 1 -- 172.21.15.28:0/1894443508 wait complete. 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.440+0000 7ff65e470740 1 -- start start 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.440+0000 7ff65e470740 1 Processor -- start 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.441+0000 7ff65e470740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393397df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 0x55d393398330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff655c98640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff655c98640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:50592/0 (socket says 172.21.15.28:50592) 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d393153700 con 0x55d392fc4390 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d39328d160 con 0x55d392fbc1b0 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d39339aaf0 con 0x55d392fe5070 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff655c98640 1 -- 172.21.15.28:0/2047393210 learned_addr learned my addr 172.21.15.28:0/2047393210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff65e470740 10 client.0 _ll_register_callbacks cb 0x55d3932a69e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.442+0000 7ff656499640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 0x55d393398330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655497640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393397df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655c98640 1 -- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 msgr2=0x55d393398330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655c98640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 0x55d393398330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655c98640 1 -- 172.21.15.28:0/2047393210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 msgr2=0x55d393397df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655c98640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393397df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff655c98640 1 -- 172.21.15.28:0/2047393210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3932bac90 con 0x55d392fbc1b0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.443+0000 7ff656499640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 0x55d393398330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.444+0000 7ff655c98640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff6480116c0 tx=0x7ff648004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a445473af31a3d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.444+0000 7ff6397fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.444+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64801e830 con 0x55d392fbc1b0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.444+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7ff64801e9d0 con 0x55d392fbc1b0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.445+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6400060d0 con 0x55d392fbc1b0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.445+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff64801ebd0 con 0x55d392fbc1b0 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.445+0000 7ff65e470740 10 client.26936 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.445+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d3933a6820 con 0x55d392fbc1b0 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6400060d0 con 0x55d392fbc1b0 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff648027070 con 0x55d392fbc1b0 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 10 client.26936 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 10 client.26936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 20 client.26936 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fa39f0 0x55d3933aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 4 client.26936 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d393304df0 con 0x55d392fa39f0 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.446+0000 7ff655497640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fa39f0 0x55d3933aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.447+0000 7ff6397fa640 10 client.26936 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.447+0000 7ff655497640 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fa39f0 0x55d3933aa210 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b6b0da63505d97 server_cookie=f47eacf5ff23ae09 in_seq=0 out_seq=0 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.448+0000 7ff6397fa640 1 -- 172.21.15.28:0/2047393210 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d393304df0 con 0x55d392fa39f0 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.448+0000 7ff6397fa640 10 client.26936 handle_command_reply: tid=0 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.448+0000 7ff65e470740 1 client.26936 shutdown 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.448+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 msgr2=0x55d3933978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.448+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff6480116c0 tx=0x7ff648004a40 comp rx=0 tx=0).stop 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 shutdown_connections 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fa39f0 0x55d3933aa210 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fe5070 0x55d393398330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff6397fa640 0 client.26936 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393397df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 --2- 172.21.15.28:0/2047393210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fbc1b0 0x55d3933978b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 >> 172.21.15.28:0/2047393210 conn(0x55d392fd9460 msgr2=0x55d39338f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 shutdown_connections 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 1 -- 172.21.15.28:0/2047393210 wait complete. 2024-09-21T05:55:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.449+0000 7ff65e470740 20 client.26936 trim_cache size 0 max 16384 2024-09-21T05:55:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.470+0000 7ff65e470740 1 Processor -- start 2024-09-21T05:55:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.470+0000 7ff65e470740 1 -- start start 2024-09-21T05:55:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d393394fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393398dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501cf6a0 con 0x55d392fc4390 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501d1300 con 0x55d392fa39f0 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff65e470740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501cfdd0 con 0x55d392fbc1b0 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:50600/0 (socket says 172.21.15.28:50600) 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 -- 172.21.15.28:0/1124205145 learned_addr learned my addr 172.21.15.28:0/1124205145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655497640 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d393394fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff656499640 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393398dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 -- 172.21.15.28:0/1124205145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 msgr2=0x55d393394fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d393394fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 -- 172.21.15.28:0/1124205145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 msgr2=0x55d393398dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393398dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 -- 172.21.15.28:0/1124205145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d393384460 con 0x55d392fa39f0 2024-09-21T05:55:06.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.471+0000 7ff655c98640 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff648005d50 tx=0x7ff648010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a491d44433f5db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.472+0000 7ff63b7fe640 1 -- 172.21.15.28:0/1124205145 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648031a00 con 0x55d392fa39f0 2024-09-21T05:55:06.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.472+0000 7ff63b7fe640 1 -- 172.21.15.28:0/1124205145 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff648003970 con 0x55d392fa39f0 2024-09-21T05:55:06.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.472+0000 7ff63b7fe640 1 -- 172.21.15.28:0/1124205145 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648004420 con 0x55d392fa39f0 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- 172.21.15.28:0/1124205145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 msgr2=0x55d393394aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff648005d50 tx=0x7ff648010940 comp rx=0 tx=0).stop 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- 172.21.15.28:0/1124205145 shutdown_connections 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fc4390 0x55d393398dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fbc1b0 0x55d393394fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 --2- 172.21.15.28:0/1124205145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fa39f0 0x55d393394aa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- 172.21.15.28:0/1124205145 >> 172.21.15.28:0/1124205145 conn(0x55d392fd9460 msgr2=0x55d393389ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- 172.21.15.28:0/1124205145 shutdown_connections 2024-09-21T05:55:06.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- 172.21.15.28:0/1124205145 wait complete. 2024-09-21T05:55:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 -- start start 2024-09-21T05:55:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.473+0000 7ff65e470740 1 Processor -- start 2024-09-21T05:55:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 0x55d39339b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fbc1b0 0x55d39339bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39340cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff655c98640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 0x55d39339b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501d0c00 con 0x55d392fbc1b0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501d04d0 con 0x55d392fc4390 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6501cd6a0 con 0x55d392fa39f0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff655c98640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 0x55d39339b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:38162/0 (socket says 172.21.15.28:38162) 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff655c98640 1 -- 172.21.15.28:0/622401767 learned_addr learned my addr 172.21.15.28:0/622401767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff65e470740 10 client.0 _ll_register_callbacks cb 0x7ff64801a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff655497640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fbc1b0 0x55d39339bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.474+0000 7ff656499640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39340cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 -- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 msgr2=0x55d39339b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 0x55d39339b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 -- 172.21.15.28:0/622401767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fbc1b0 msgr2=0x55d39339bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fbc1b0 0x55d39339bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 -- 172.21.15.28:0/622401767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d39340d200 con 0x55d392fc4390 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff656499640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39340cb90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff64c104560 tx=0x7ff64c0d65b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6155eb3b1a36dc24 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff6397fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c005200 con 0x55d392fc4390 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff648003970 con 0x55d392fc4390 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7ff64c004250 con 0x55d392fc4390 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff64c0cc770 con 0x55d392fc4390 2024-09-21T05:55:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff65e470740 10 client.26942 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d393414560 con 0x55d392fc4390 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.475+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff648003970 con 0x55d392fc4390 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7ff64c03a030 con 0x55d392fc4390 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 10 client.26942 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 10 client.26942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 20 client.26942 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fe5070 0x55d393416d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 4 client.26942 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3933008a0 con 0x55d392fe5070 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff655c98640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fe5070 0x55d393416d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff655c98640 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fe5070 0x55d393416d70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a37ff535c10a1f server_cookie=4c1ff42edb31f745 in_seq=0 out_seq=0 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.476+0000 7ff6397fa640 10 client.26942 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff6397fa640 1 -- 172.21.15.28:0/622401767 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3933008a0 con 0x55d392fe5070 2024-09-21T05:55:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff6397fa640 10 client.26942 handle_command_reply: tid=0 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 client.26942 shutdown 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 msgr2=0x55d39340cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39340cb90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff64c104560 tx=0x7ff64c0d65b0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 shutdown_connections 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d392fe5070 0x55d393416d70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d392fc4390 0x55d39340cb90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d392fbc1b0 0x55d39339bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 --2- 172.21.15.28:0/622401767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d392fa39f0 0x55d39339b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 >> 172.21.15.28:0/622401767 conn(0x55d392fd9460 msgr2=0x55d39338d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff6397fa640 0 client.26942 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 shutdown_connections 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 1 -- 172.21.15.28:0/622401767 wait complete. 2024-09-21T05:55:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.477+0000 7ff65e470740 20 client.26942 trim_cache size 0 max 16384 2024-09-21T05:55:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 msgr2=0x7ff63c0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 0x7ff63c0b8d60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55d39338af40 tx=0x7ff64801a5d0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 msgr2=0x7ff65010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 0x7ff65010aa50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff64c035040 tx=0x7ff64c033b30 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 shutdown_connections 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650108cf0 0x7ff65010aa50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6501083a0 0x7ff65010a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff63c0b6850 0x7ff63c0b8d60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 --2- 172.21.15.28:0/3389809007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff650102a60 0x7ff65010f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 >> 172.21.15.28:0/3389809007 conn(0x7ff650079730 msgr2=0x7ff65006b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 shutdown_connections 2024-09-21T05:55:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:06.478+0000 7ff657efc640 1 -- 172.21.15.28:0/3389809007 wait complete. 2024-09-21T05:55:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:06 smithi129 ceph-mon[34885]: pgmap v569: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:55:06.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:06 smithi181 ceph-mon[34819]: pgmap v569: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:55:06.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:06 smithi028 ceph-mon[31227]: pgmap v569: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:55:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:07 smithi028 ceph-mon[31227]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T05:55:07.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:07 smithi129 ceph-mon[34885]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T05:55:07.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:07 smithi181 ceph-mon[34819]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T05:55:08.348 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:55:08.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:55:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:08.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:08 smithi129 ceph-mon[34885]: pgmap v571: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:08.734 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:08 smithi028 ceph-mon[31227]: pgmap v571: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:08 smithi181 ceph-mon[34819]: pgmap v571: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/466886796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4103b60 msgr2=0x7f8bd4103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/466886796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4103b60 0x7f8bd4103f70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009900 tx=0x7f8bcc02a410 comp rx=0 tx=0).stop 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/466886796 shutdown_connections 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/466886796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd4109df0 0x7f8bd4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/466886796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd41094a0 0x7f8bd41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/466886796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4103b60 0x7f8bd4103f70 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/466886796 >> 172.21.15.28:0/466886796 conn(0x7f8bd407b850 msgr2=0x7f8bd407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/466886796 shutdown_connections 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.117+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/466886796 wait complete. 2024-09-21T05:55:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 Processor -- start 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 -- start start 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd4103b60 0x7f8bd419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd41094a0 0x7f8bd419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4113ab0 con 0x7f8bd4109df0 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4113930 con 0x7f8bd4103b60 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdc5ef640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4113c30 con 0x7f8bd41094a0 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46806/0 (socket says 172.21.15.28:46806) 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3454288021 learned_addr learned my addr 172.21.15.28:0/3454288021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd41094a0 0x7f8bd419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.118+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd4103b60 0x7f8bd419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3454288021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd41094a0 msgr2=0x7f8bd419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd41094a0 0x7f8bd419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3454288021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd4103b60 msgr2=0x7f8bd419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd4103b60 0x7f8bd419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3454288021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd41a0350 con 0x7f8bd4109df0 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400eaa0 tx=0x7f8bc400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bd37fe640 1 -- 172.21.15.28:0/3454288021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4004d20 con 0x7f8bd4109df0 2024-09-21T05:55:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd41a0640 con 0x7f8bd4109df0 2024-09-21T05:55:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bd37fe640 1 -- 172.21.15.28:0/3454288021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc4005020 con 0x7f8bd4109df0 2024-09-21T05:55:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bd37fe640 1 -- 172.21.15.28:0/3454288021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc40116b0 con 0x7f8bd4109df0 2024-09-21T05:55:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.119+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd41a6e60 con 0x7f8bd4109df0 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.122+0000 7f8be2b63740 1 Processor -- start 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.122+0000 7f8bd37fe640 1 -- 172.21.15.28:0/3454288021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8bc40051d0 con 0x7f8bd4109df0 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.122+0000 7f8be2b63740 1 -- start start 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d9809350 0x55f1d9bb0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e8670 0x55f1d9bb0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9b26e60 con 0x55f1d97e0490 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9b16290 con 0x55f1d97e8670 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8be2b63740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9b1a0a0 con 0x55f1d9809350 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46810/0 (socket says 172.21.15.28:46810) 2024-09-21T05:55:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 -- 172.21.15.28:0/3138521779 learned_addr learned my addr 172.21.15.28:0/3138521779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e8670 0x55f1d9bb0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d9809350 0x55f1d9bb0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 -- 172.21.15.28:0/3138521779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d9809350 msgr2=0x55f1d9bb0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d9809350 0x55f1d9bb0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 -- 172.21.15.28:0/3138521779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e8670 msgr2=0x55f1d9bb0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e8670 0x55f1d9bb0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 -- 172.21.15.28:0/3138521779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d9874930 con 0x55f1d97e0490 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bd37fe640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 0x7f8bbc0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc02a930 tx=0x7f8bcc032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e47f820f4e1d9f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.123+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 0x7f8bbc0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.124+0000 7f8bd37fe640 1 -- 172.21.15.28:0/3454288021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f8bc40e7960 con 0x7f8bd4109df0 2024-09-21T05:55:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.124+0000 7f8bd0ff9640 1 -- 172.21.15.28:0/3138521779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc03f070 con 0x55f1d97e0490 2024-09-21T05:55:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.124+0000 7f8bd0ff9640 1 -- 172.21.15.28:0/3138521779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bcc0316a0 con 0x55f1d97e0490 2024-09-21T05:55:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.124+0000 7f8bd0ff9640 1 -- 172.21.15.28:0/3138521779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc035070 con 0x55f1d97e0490 2024-09-21T05:55:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- 172.21.15.28:0/3138521779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 msgr2=0x55f1d9bb0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 0x7f8bbc0b8bc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4106530 tx=0x7f8bc8013040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc02a930 tx=0x7f8bcc032ce0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- 172.21.15.28:0/3138521779 shutdown_connections 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e8670 0x55f1d9bb0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e0490 0x55f1d9bb0970 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3138521779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d9809350 0x55f1d9bb0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- 172.21.15.28:0/3138521779 >> 172.21.15.28:0/3138521779 conn(0x55f1d97fd740 msgr2=0x55f1d9bae320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- 172.21.15.28:0/3138521779 shutdown_connections 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- 172.21.15.28:0/3138521779 wait complete. 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 -- start start 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.125+0000 7f8be2b63740 1 Processor -- start 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bbbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bbc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9977890 con 0x55f1d9809350 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9ab0d40 con 0x55f1d97e0490 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f1d9bbebb0 con 0x55f1d97e8670 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8be2b63740 10 client.0 _ll_register_callbacks cb 0x55f1d9b14c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46820/0 (socket says 172.21.15.28:46820) 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3754202177 learned_addr learned my addr 172.21.15.28:0/3754202177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bbc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bbbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 msgr2=0x55f1d9bbc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bbc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3754202177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 msgr2=0x55f1d9bbbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bbbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 -- 172.21.15.28:0/3754202177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d9bc8490 con 0x55f1d9809350 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bbc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bbbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40a9d50 tx=0x7f8bc40ad1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cef3862a702097c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.126+0000 7f8bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc401f030 con 0x55f1d9809350 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f8bc40a90d0 con 0x55f1d9809350 2024-09-21T05:55:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bcc0316a0 con 0x55f1d9809350 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8bc40b2260 con 0x55f1d9809350 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 10 client.17418 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f1d9bcae00 con 0x55f1d9809350 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bcc0316a0 con 0x55f1d9809350 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f8bc400f040 con 0x55f1d9809350 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 10 client.17418 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 20 client.17418 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d97c7cd0 0x55f1d9bce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 4 client.17418 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.127+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d9b28db0 con 0x55f1d97c7cd0 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.128+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d97c7cd0 0x55f1d9bce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.128+0000 7f8bb9ffb640 10 client.17418 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.128+0000 7f8bda38b640 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d97c7cd0 0x55f1d9bce300 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b235d5af1220cb server_cookie=a7b31d2421b2919d in_seq=0 out_seq=0 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/3754202177 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f1d9b28db0 con 0x55f1d97c7cd0 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8bb9ffb640 10 client.17418 handle_command_reply: tid=0 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 client.17418 shutdown 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 msgr2=0x55f1d9bc7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40a9d50 tx=0x7f8bc40ad1a0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 shutdown_connections 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d97c7cd0 0x55f1d9bce300 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9bc7ea0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bbc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8bb9ffb640 0 client.17418 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 --2- 172.21.15.28:0/3754202177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bbbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 >> 172.21.15.28:0/3754202177 conn(0x55f1d97fd740 msgr2=0x55f1d9bb35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 shutdown_connections 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 1 -- 172.21.15.28:0/3754202177 wait complete. 2024-09-21T05:55:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.129+0000 7f8be2b63740 20 client.17418 trim_cache size 0 max 16384 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 Processor -- start 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 -- start start 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e0490 0x55f1d9bc7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d9809350 0x55f1d9bb8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41af770 con 0x55f1d97e8670 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41b13d0 con 0x55f1d9809350 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8be2b63740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41afea0 con 0x55f1d97e0490 2024-09-21T05:55:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8bd9b8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8bd9b8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46830/0 (socket says 172.21.15.28:46830) 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.147+0000 7f8bd9b8a640 1 -- 172.21.15.28:0/1009567062 learned_addr learned my addr 172.21.15.28:0/1009567062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bda38b640 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e0490 0x55f1d9bc7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d9809350 0x55f1d9bb8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 -- 172.21.15.28:0/1009567062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e0490 msgr2=0x55f1d9bc7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e0490 0x55f1d9bc7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 -- 172.21.15.28:0/1009567062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d9809350 msgr2=0x55f1d9bb8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d9809350 0x55f1d9bb8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 -- 172.21.15.28:0/1009567062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d9bb3290 con 0x55f1d97e8670 2024-09-21T05:55:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80217f0 tx=0x7f8bc800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ce26b85f6b08af server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bbbfff640 1 -- 172.21.15.28:0/1009567062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc801fb50 con 0x55f1d97e8670 2024-09-21T05:55:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bbbfff640 1 -- 172.21.15.28:0/1009567062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc8021300 con 0x55f1d97e8670 2024-09-21T05:55:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.148+0000 7f8bbbfff640 1 -- 172.21.15.28:0/1009567062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80206a0 con 0x55f1d97e8670 2024-09-21T05:55:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- 172.21.15.28:0/1009567062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 msgr2=0x55f1d9bb6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80217f0 tx=0x7f8bc800a2d0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- 172.21.15.28:0/1009567062 shutdown_connections 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d9809350 0x55f1d9bb8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d97e8670 0x55f1d9bb6550 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 --2- 172.21.15.28:0/1009567062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e0490 0x55f1d9bc7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- 172.21.15.28:0/1009567062 >> 172.21.15.28:0/1009567062 conn(0x55f1d97fd740 msgr2=0x55f1d9bba670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- 172.21.15.28:0/1009567062 shutdown_connections 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- 172.21.15.28:0/1009567062 wait complete. 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.149+0000 7f8be2b63740 1 -- start start 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 Processor -- start 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bc0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bc0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41b0cd0 con 0x55f1d9809350 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41b05a0 con 0x55f1d97e0490 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41ad770 con 0x55f1d97e8670 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8be2b63740 10 client.0 _ll_register_callbacks cb 0x7f8bc4004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46840/0 (socket says 172.21.15.28:46840) 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 -- 172.21.15.28:0/2919622545 learned_addr learned my addr 172.21.15.28:0/2919622545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bd9b8a640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bc0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bda38b640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bc0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 -- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 msgr2=0x55f1d9bc0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bc0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 -- 172.21.15.28:0/2919622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 msgr2=0x55f1d9bc0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bc0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.150+0000 7f8bdab8c640 1 -- 172.21.15.28:0/2919622545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d9c36710 con 0x55f1d9809350 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bdab8c640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40aa9f0 tx=0x7f8bc40ace00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67766013fb63b427 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc401f070 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f8bc40b33f0 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bc8021300 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8bc40ac450 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 10 client.17424 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f1d9c3da20 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc8021300 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f8bc40b35f0 con 0x55f1d9809350 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 10 client.17424 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 20 client.17424 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d979d080 0x55f1d9c401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 4 client.17424 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:55:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.151+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1d9b24860 con 0x55f1d979d080 2024-09-21T05:55:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.152+0000 7f8bda38b640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d979d080 0x55f1d9c401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.152+0000 7f8bda38b640 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d979d080 0x55f1d9c401e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2beeaf09d94a68 server_cookie=d4b99148f2ffc4cb in_seq=0 out_seq=0 2024-09-21T05:55:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.152+0000 7f8bb9ffb640 10 client.17424 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8bb9ffb640 1 -- 172.21.15.28:0/2919622545 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1d9b24860 con 0x55f1d979d080 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8bb9ffb640 10 client.17424 handle_command_reply: tid=0 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 client.17424 shutdown 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 msgr2=0x55f1d9c360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40aa9f0 tx=0x7f8bc40ace00 comp rx=0 tx=0).stop 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 shutdown_connections 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55f1d979d080 0x55f1d9c401e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d9809350 0x55f1d9c360a0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f1d97e8670 0x55f1d9bc0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 --2- 172.21.15.28:0/2919622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f1d97e0490 0x55f1d9bc0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 >> 172.21.15.28:0/2919622545 conn(0x55f1d97fd740 msgr2=0x55f1d9bb1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8bb9ffb640 0 client.17424 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 shutdown_connections 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 1 -- 172.21.15.28:0/2919622545 wait complete. 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.154+0000 7f8be2b63740 20 client.17424 trim_cache size 0 max 16384 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2319b9a4-11f3-496b-9fc8-8eb7f3571f92", 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:55:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 msgr2=0x7f8bbc0b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 0x7f8bbc0b8bc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4106530 tx=0x7f8bc8013040 comp rx=0 tx=0).stop 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 msgr2=0x7f8bd419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400eaa0 tx=0x7f8bc400eea0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 shutdown_connections 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8bbc0b66b0 0x7f8bbc0b8bc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4109df0 0x7f8bd419fce0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8bd41094a0 0x7f8bd419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 --2- 172.21.15.28:0/3454288021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bd4103b60 0x7f8bd419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:09.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 >> 172.21.15.28:0/3454288021 conn(0x7f8bd407b850 msgr2=0x7f8bd4107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:09.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 shutdown_connections 2024-09-21T05:55:09.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:09.155+0000 7f8bdc5ef640 1 -- 172.21.15.28:0/3454288021 wait complete. 2024-09-21T05:55:10.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:55:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:10 smithi129 ceph-mon[34885]: pgmap v572: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:10 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:55:10.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:10 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:10 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:55:10.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:10 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:10 smithi028 ceph-mon[31227]: pgmap v572: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:10 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:55:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:10 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:10 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:55:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:10 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:10 smithi181 ceph-mon[34819]: pgmap v572: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T05:55:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:10 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:55:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:10 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:10 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:55:10.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:10 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.841+0000 7f8727ea5640 1 -- 172.21.15.28:0/3511124727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 msgr2=0x7f872010d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.841+0000 7f8727ea5640 1 --2- 172.21.15.28:0/3511124727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f872010d860 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8714009ca0 tx=0x7f871402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 -- 172.21.15.28:0/3511124727 shutdown_connections 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 --2- 172.21.15.28:0/3511124727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f872010de20 0x7f8720110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 --2- 172.21.15.28:0/3511124727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f872010d860 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 --2- 172.21.15.28:0/3511124727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87201049c0 0x7f8720104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 -- 172.21.15.28:0/3511124727 >> 172.21.15.28:0/3511124727 conn(0x7f8720079710 msgr2=0x7f8720079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 -- 172.21.15.28:0/3511124727 shutdown_connections 2024-09-21T05:55:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 -- 172.21.15.28:0/3511124727 wait complete. 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 Processor -- start 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.842+0000 7f8727ea5640 1 -- start start 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87201049c0 0x7f8720102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872010de20 0x7f8720103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113a80 con 0x7f8720105310 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113900 con 0x7f87201049c0 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8727ea5640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113c00 con 0x7f872010de20 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46844/0 (socket says 172.21.15.28:46844) 2024-09-21T05:55:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 -- 172.21.15.28:0/2510017853 learned_addr learned my addr 172.21.15.28:0/2510017853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8726442640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872010de20 0x7f8720103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725c41640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87201049c0 0x7f8720102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 -- 172.21.15.28:0/2510017853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872010de20 msgr2=0x7f8720103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872010de20 0x7f8720103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 -- 172.21.15.28:0/2510017853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87201049c0 msgr2=0x7f8720102700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87201049c0 0x7f8720102700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.843+0000 7f8725440640 1 -- 172.21.15.28:0/2510017853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87201a8770 con 0x7f8720105310 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f8725440640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f87140319f0 tx=0x7f8714031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f871effd640 1 -- 172.21.15.28:0/2510017853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714003bf0 con 0x7f8720105310 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87201a8a00 con 0x7f8720105310 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f871effd640 1 -- 172.21.15.28:0/2510017853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8714003d90 con 0x7f8720105310 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f871effd640 1 -- 172.21.15.28:0/2510017853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871402fb70 con 0x7f8720105310 2024-09-21T05:55:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.844+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87201a8f40 con 0x7f8720105310 2024-09-21T05:55:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.847+0000 7f872e419740 1 Processor -- start 2024-09-21T05:55:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.847+0000 7f872e419740 1 -- start start 2024-09-21T05:55:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.847+0000 7f872e419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.847+0000 7f872e419740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e416ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f872e419740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e416eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f872e419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e41660e50 con 0x556e41343070 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f872e419740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556e41650280 con 0x556e41322390 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f872e419740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556e41654090 con 0x556e4131a1b0 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f871effd640 1 -- 172.21.15.28:0/2510017853 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f871402fd10 con 0x7f8720105310 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46858/0 (socket says 172.21.15.28:46858) 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 -- 172.21.15.28:0/3744669875 learned_addr learned my addr 172.21.15.28:0/3744669875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8725c41640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e416ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8725440640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e416eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 -- 172.21.15.28:0/3744669875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 msgr2=0x556e416ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e416ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 -- 172.21.15.28:0/3744669875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 msgr2=0x556e416eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e416eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 -- 172.21.15.28:0/3744669875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e413ae650 con 0x556e41343070 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8725c41640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e416ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8726442640 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f871800b700 tx=0x7f871800bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691134fc2d17f8dc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.848+0000 7f8707fff640 1 -- 172.21.15.28:0/3744669875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718003dd0 con 0x556e41343070 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.849+0000 7f8707fff640 1 -- 172.21.15.28:0/3744669875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8718015070 con 0x556e41343070 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.849+0000 7f8707fff640 1 -- 172.21.15.28:0/3744669875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87180106a0 con 0x556e41343070 2024-09-21T05:55:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.849+0000 7f871effd640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 0x7f87080b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.849+0000 7f8725c41640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 0x7f87080b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.849+0000 7f871effd640 1 -- 172.21.15.28:0/2510017853 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f87141089f0 con 0x7f8720105310 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- 172.21.15.28:0/3744669875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 msgr2=0x556e416ea270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f8725c41640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 0x7f87080b8b90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x556e416e8ef0 tx=0x7f871000a250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f871800b700 tx=0x7f871800bbc0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- 172.21.15.28:0/3744669875 shutdown_connections 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e416eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e416ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 --2- 172.21.15.28:0/3744669875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e416ea270 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- 172.21.15.28:0/3744669875 >> 172.21.15.28:0/3744669875 conn(0x556e41337460 msgr2=0x556e416e8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- 172.21.15.28:0/3744669875 shutdown_connections 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- 172.21.15.28:0/3744669875 wait complete. 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 -- start start 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.850+0000 7f872e419740 1 Processor -- start 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e4131a1b0 0x556e416f5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41322390 0x556e416f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e414b1700 con 0x556e41343070 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556e415eb110 con 0x556e4131a1b0 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556e416f8a50 con 0x556e41322390 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f872e419740 10 client.0 _ll_register_callbacks cb 0x556e41604990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46862/0 (socket says 172.21.15.28:46862) 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 -- 172.21.15.28:0/2365992809 learned_addr learned my addr 172.21.15.28:0/2365992809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8725c41640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e4131a1b0 0x556e416f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8725440640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41322390 0x556e416f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 -- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41322390 msgr2=0x556e416f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41322390 0x556e416f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 -- 172.21.15.28:0/2365992809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e4131a1b0 msgr2=0x556e416f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e4131a1b0 0x556e416f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 -- 172.21.15.28:0/2365992809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e41618c40 con 0x556e41343070 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f8726442640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8718000c00 tx=0x7f8718003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a83cacf7d7114 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.851+0000 7f87057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871802a060 con 0x556e41343070 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f8718005440 con 0x556e41343070 2024-09-21T05:55:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8718015070 con 0x556e41343070 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8718011440 con 0x556e41343070 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 10 client.17436 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x556e417047c0 con 0x556e41343070 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8718015070 con 0x556e41343070 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f8718004be0 con 0x556e41343070 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 10 client.17436 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 10 client.17436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 20 client.17436 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e413019f0 0x556e41707bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 4 client.17436 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.852+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e41662da0 con 0x556e413019f0 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.853+0000 7f8725c41640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e413019f0 0x556e41707bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.853+0000 7f87057fa640 10 client.17436 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:10.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.853+0000 7f8725c41640 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e413019f0 0x556e41707bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f96878f08fde35 server_cookie=3e6b00a7cc44ebe3 in_seq=0 out_seq=0 2024-09-21T05:55:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.854+0000 7f87057fa640 1 -- 172.21.15.28:0/2365992809 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x556e41662da0 con 0x556e413019f0 2024-09-21T05:55:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.854+0000 7f87057fa640 10 client.17436 handle_command_reply: tid=0 2024-09-21T05:55:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.854+0000 7f872e419740 1 client.17436 shutdown 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.854+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 msgr2=0x556e41701da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.854+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8718000c00 tx=0x7f8718003e00 comp rx=0 tx=0).stop 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 shutdown_connections 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e413019f0 0x556e41707bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41343070 0x556e41701da0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41322390 0x556e416f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 --2- 172.21.15.28:0/2365992809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e4131a1b0 0x556e416f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f87057fa640 0 client.17436 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 >> 172.21.15.28:0/2365992809 conn(0x556e41337460 msgr2=0x556e416ed450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 shutdown_connections 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 1 -- 172.21.15.28:0/2365992809 wait complete. 2024-09-21T05:55:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.855+0000 7f872e419740 20 client.17436 trim_cache size 0 max 16384 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 Processor -- start 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 -- start start 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e41701d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41343070 0x556e416e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.872+0000 7f872e419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b1a20 con 0x556e4131a1b0 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f872e419740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b3680 con 0x556e41322390 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f872e419740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b2150 con 0x556e41343070 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46868/0 (socket says 172.21.15.28:46868) 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 -- 172.21.15.28:0/3197089367 learned_addr learned my addr 172.21.15.28:0/3197089367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725440640 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e41701d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8726442640 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41343070 0x556e416e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 -- 172.21.15.28:0/3197089367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41343070 msgr2=0x556e416e7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41343070 0x556e416e7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 -- 172.21.15.28:0/3197089367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 msgr2=0x556e41701d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e41701d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 -- 172.21.15.28:0/3197089367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e416ec790 con 0x556e4131a1b0 2024-09-21T05:55:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f8725c41640 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f87100058c0 tx=0x7f8710015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999a1dc043c8843f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f87077fe640 1 -- 172.21.15.28:0/3197089367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710004020 con 0x556e4131a1b0 2024-09-21T05:55:10.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f87077fe640 1 -- 172.21.15.28:0/3197089367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87100041c0 con 0x556e4131a1b0 2024-09-21T05:55:10.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.873+0000 7f87077fe640 1 -- 172.21.15.28:0/3197089367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87100106a0 con 0x556e4131a1b0 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- 172.21.15.28:0/3197089367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 msgr2=0x556e416eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f87100058c0 tx=0x7f8710015000 comp rx=0 tx=0).stop 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- 172.21.15.28:0/3197089367 shutdown_connections 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e41343070 0x556e416e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41322390 0x556e41701d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 --2- 172.21.15.28:0/3197089367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e4131a1b0 0x556e416eda80 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- 172.21.15.28:0/3197089367 >> 172.21.15.28:0/3197089367 conn(0x556e41337460 msgr2=0x556e416ed370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- 172.21.15.28:0/3197089367 shutdown_connections 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- 172.21.15.28:0/3197089367 wait complete. 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.874+0000 7f872e419740 1 -- start start 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 Processor -- start 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e4170b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41343070 0x556e41708e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b2f80 con 0x556e41322390 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87201b2850 con 0x556e41343070 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f87201afa20 con 0x556e4131a1b0 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f872e419740 10 client.0 _ll_register_callbacks cb 0x7f8718004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f8725440640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f8725440640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46874/0 (socket says 172.21.15.28:46874) 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f8725440640 1 -- 172.21.15.28:0/1417071473 learned_addr learned my addr 172.21.15.28:0/1417071473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f8725c41640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e4170b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.875+0000 7f8726442640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41343070 0x556e41708e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 -- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 msgr2=0x556e4170b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e4170b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 -- 172.21.15.28:0/1417071473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41343070 msgr2=0x556e41708e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41343070 0x556e41708e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 -- 172.21.15.28:0/1417071473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e417095f0 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f8725440640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f87140ce7a0 tx=0x7f871402f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a5a5ae33aa9315 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140cdf80 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) ==== 8572+0+0 (secure 0 0 0) 0x7f87140cfb40 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87100041c0 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f87140ce320 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f872e419740 10 client.17442 fetch_fsmap learned FSMap version 47 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x556e41772280 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87100041c0 con 0x556e41322390 2024-09-21T05:55:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.876+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8390+0+0 (secure 0 0 0) 0x7f87140cd4e0 con 0x556e41322390 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 10 client.17442 fetch_fsmap finished waiting for FSMap version 47 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 20 client.17442 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e412d6da0 0x556e41774a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 4 client.17442 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556e4165e850 con 0x556e412d6da0 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f8725c41640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e412d6da0 0x556e41774a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f87057fa640 10 client.17442 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.877+0000 7f8725c41640 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e412d6da0 0x556e41774a70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328fd22db6210276 server_cookie=951decffeaefe680 in_seq=0 out_seq=0 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f87057fa640 1 -- 172.21.15.28:0/1417071473 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3694 (crc 0 0 0) 0x556e4165e850 con 0x556e412d6da0 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f87057fa640 10 client.17442 handle_command_reply: tid=0 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 client.17442 shutdown 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 msgr2=0x556e41708940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f87140ce7a0 tx=0x7f871402f660 comp rx=0 tx=0).stop 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 shutdown_connections 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x556e412d6da0 0x556e41774a70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556e41343070 0x556e41708e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e41322390 0x556e41708940 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 --2- 172.21.15.28:0/1417071473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556e4131a1b0 0x556e4170b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f87057fa640 0 client.17442 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 >> 172.21.15.28:0/1417071473 conn(0x556e41337460 msgr2=0x556e416e7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 shutdown_connections 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 1 -- 172.21.15.28:0/1417071473 wait complete. 2024-09-21T05:55:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.878+0000 7f872e419740 20 client.17442 trim_cache size 0 max 16384 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2319b9a4-11f3-496b-9fc8-8eb7f3571f92", 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:55:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:09.151907+0000", 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:10.892 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 msgr2=0x7f87080b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 0x7f87080b8b90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x556e416e8ef0 tx=0x7f871000a250 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 msgr2=0x7f8720102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f87140319f0 tx=0x7f8714031a20 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 shutdown_connections 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f872010de20 0x7f8720103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f87080b6680 0x7f87080b8b90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8720105310 0x7f8720102c40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 --2- 172.21.15.28:0/2510017853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87201049c0 0x7f8720102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 >> 172.21.15.28:0/2510017853 conn(0x7f8720079710 msgr2=0x7f8720106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 shutdown_connections 2024-09-21T05:55:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:10.879+0000 7f8727ea5640 1 -- 172.21.15.28:0/2510017853 wait complete. 2024-09-21T05:55:11.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:11 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:55:11.601 INFO:tasks.cephfs.filesystem:scrub status for tag:2319b9a4-11f3-496b-9fc8-8eb7f3571f92 - {'path': '/', 'tag': '2319b9a4-11f3-496b-9fc8-8eb7f3571f92', 'options': 'recursive,force'} 2024-09-21T05:55:11.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:11 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:55:11.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:11 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:55:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:12 smithi129 ceph-mon[34885]: pgmap v573: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:55:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:12 smithi181 ceph-mon[34819]: pgmap v573: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:55:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:12 smithi028 ceph-mon[31227]: pgmap v573: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:55:14.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:14 smithi129 ceph-mon[34885]: pgmap v574: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:55:14.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:14 smithi181 ceph-mon[34819]: pgmap v574: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:55:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:14 smithi028 ceph-mon[31227]: pgmap v574: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:55:16.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:16 smithi129 ceph-mon[34885]: pgmap v575: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:55:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:16 smithi181 ceph-mon[34819]: pgmap v575: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:55:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:16 smithi028 ceph-mon[31227]: pgmap v575: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:55:18.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:18 smithi181 ceph-mon[34819]: pgmap v576: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 11 op/s 2024-09-21T05:55:18.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:18 smithi181 ceph-mon[34819]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T05:55:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:18 smithi028 ceph-mon[31227]: pgmap v576: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 11 op/s 2024-09-21T05:55:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:18 smithi028 ceph-mon[31227]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T05:55:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:18 smithi129 ceph-mon[34885]: pgmap v576: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 11 op/s 2024-09-21T05:55:18.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:18 smithi129 ceph-mon[34885]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T05:55:20.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:20 smithi181 ceph-mon[34819]: pgmap v578: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-21T05:55:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:20 smithi028 ceph-mon[31227]: pgmap v578: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-21T05:55:20.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:20 smithi129 ceph-mon[34885]: pgmap v578: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-21T05:55:21.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:21 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:55:21.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:21 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:55:21.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:21 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:55:22.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:22 smithi181 ceph-mon[34819]: pgmap v579: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T05:55:22.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:22 smithi028 ceph-mon[31227]: pgmap v579: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T05:55:22.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:22 smithi129 ceph-mon[34885]: pgmap v579: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T05:55:24.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:24 smithi181 ceph-mon[34819]: pgmap v580: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T05:55:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:24 smithi028 ceph-mon[31227]: pgmap v580: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T05:55:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:24 smithi129 ceph-mon[34885]: pgmap v580: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T05:55:26.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:26 smithi181 ceph-mon[34819]: pgmap v581: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:26 smithi028 ceph-mon[31227]: pgmap v581: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:26.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:26 smithi129 ceph-mon[34885]: pgmap v581: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:28.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:28 smithi181 ceph-mon[34819]: pgmap v582: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:28 smithi028 ceph-mon[31227]: pgmap v582: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:28.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:28 smithi129 ceph-mon[34885]: pgmap v582: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:30.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:30 smithi181 ceph-mon[34819]: pgmap v583: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T05:55:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:30 smithi028 ceph-mon[31227]: pgmap v583: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T05:55:30.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:30 smithi129 ceph-mon[34885]: pgmap v583: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T05:55:32.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:32 smithi181 ceph-mon[34819]: pgmap v584: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:32 smithi028 ceph-mon[31227]: pgmap v584: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:32.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:32 smithi129 ceph-mon[34885]: pgmap v584: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:34.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:34 smithi181 ceph-mon[34819]: pgmap v585: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:34.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:34 smithi028 ceph-mon[31227]: pgmap v585: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:34.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:34 smithi129 ceph-mon[34885]: pgmap v585: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:36.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:36 smithi181 ceph-mon[34819]: pgmap v586: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:36.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:36 smithi028 ceph-mon[31227]: pgmap v586: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:36.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:36 smithi129 ceph-mon[34885]: pgmap v586: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:38.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:38 smithi181 ceph-mon[34819]: pgmap v587: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:38.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:38 smithi028 ceph-mon[31227]: pgmap v587: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:38.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:38 smithi129 ceph-mon[34885]: pgmap v587: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:40.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:40 smithi181 ceph-mon[34819]: pgmap v588: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:40.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:40 smithi028 ceph-mon[31227]: pgmap v588: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:40.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:40 smithi129 ceph-mon[34885]: pgmap v588: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:41.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:55:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 -- 172.21.15.28:0/1467176085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 msgr2=0x7ff19c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 --2- 172.21.15.28:0/1467176085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1087b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff190009900 tx=0x7ff19002a410 comp rx=0 tx=0).stop 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 -- 172.21.15.28:0/1467176085 shutdown_connections 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 --2- 172.21.15.28:0/1467176085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c108cf0 0x7ff19c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 --2- 172.21.15.28:0/1467176085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1087b0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 --2- 172.21.15.28:0/1467176085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c102a60 0x7ff19c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 -- 172.21.15.28:0/1467176085 >> 172.21.15.28:0/1467176085 conn(0x7ff19c079730 msgr2=0x7ff19c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 -- 172.21.15.28:0/1467176085 shutdown_connections 2024-09-21T05:55:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.337+0000 7ff1a4169640 1 -- 172.21.15.28:0/1467176085 wait complete. 2024-09-21T05:55:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.338+0000 7ff1a4169640 1 Processor -- start 2024-09-21T05:55:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.338+0000 7ff1a4169640 1 -- start start 2024-09-21T05:55:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102a60 0x7ff19c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c108cf0 0x7ff19c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c112960 con 0x7ff19c1083a0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1127e0 con 0x7ff19c108cf0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a4169640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c112ae0 con 0x7ff19c102a60 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38010/0 (socket says 172.21.15.28:38010) 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 -- 172.21.15.28:0/3581287903 learned_addr learned my addr 172.21.15.28:0/3581287903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102a60 0x7ff19c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c108cf0 0x7ff19c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 -- 172.21.15.28:0/3581287903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102a60 msgr2=0x7ff19c199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102a60 0x7ff19c199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 -- 172.21.15.28:0/3581287903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c108cf0 msgr2=0x7ff19c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c108cf0 0x7ff19c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 -- 172.21.15.28:0/3581287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c19f2e0 con 0x7ff19c1083a0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.339+0000 7ff1a1704640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff190032fd0 tx=0x7ff190004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.340+0000 7ff19affd640 1 -- 172.21.15.28:0/3581287903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19003f030 con 0x7ff19c1083a0 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.340+0000 7ff19affd640 1 -- 172.21.15.28:0/3581287903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff190004490 con 0x7ff19c1083a0 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.340+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c19f570 con 0x7ff19c1083a0 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.340+0000 7ff19affd640 1 -- 172.21.15.28:0/3581287903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190035070 con 0x7ff19c1083a0 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.340+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c19fa50 con 0x7ff19c1083a0 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.342+0000 7ff1aa6dd740 1 Processor -- start 2024-09-21T05:55:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 -- start start 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff19affd640 1 -- 172.21.15.28:0/3581287903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff190042030 con 0x7ff19c1083a0 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812be1b0 0x55b88168e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812c6390 0x55b88168ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b881604e50 con 0x55b8812e7070 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b8815f4280 con 0x55b8812c6390 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b8815f8090 con 0x55b8812be1b0 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38012/0 (socket says 172.21.15.28:38012) 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 -- 172.21.15.28:0/3131027352 learned_addr learned my addr 172.21.15.28:0/3131027352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812be1b0 0x55b88168e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a1704640 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812c6390 0x55b88168ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 -- 172.21.15.28:0/3131027352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812be1b0 msgr2=0x55b88168e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812be1b0 0x55b88168e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 -- 172.21.15.28:0/3131027352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812c6390 msgr2=0x55b88168ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.343+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812c6390 0x55b88168ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff1a2706640 1 -- 172.21.15.28:0/3131027352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b881352650 con 0x55b8812e7070 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff19400cb40 tx=0x7ff19400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71448312cd1b054 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff19affd640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 0x7ff1840b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff183fff640 1 -- 172.21.15.28:0/3131027352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194005020 con 0x55b8812e7070 2024-09-21T05:55:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 0x7ff1840b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff183fff640 1 -- 172.21.15.28:0/3131027352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1940047a0 con 0x55b8812e7070 2024-09-21T05:55:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff183fff640 1 -- 172.21.15.28:0/3131027352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194010690 con 0x55b8812e7070 2024-09-21T05:55:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.344+0000 7ff19affd640 1 -- 172.21.15.28:0/3581287903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7ff1901078b0 con 0x7ff19c1083a0 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3131027352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 msgr2=0x55b88168e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 0x7ff1840b8bc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff18c005fd0 tx=0x7ff18c009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff19400cb40 tx=0x7ff19400e350 comp rx=0 tx=0).stop 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3131027352 shutdown_connections 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812c6390 0x55b88168ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812be1b0 0x55b88168e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3131027352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88168e270 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3131027352 >> 172.21.15.28:0/3131027352 conn(0x55b8812db460 msgr2=0x55b88168c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3131027352 shutdown_connections 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.345+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3131027352 wait complete. 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 -- start start 2024-09-21T05:55:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 Processor -- start 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b88169ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b88169bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b881455700 con 0x55b8812e7070 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b88158f110 con 0x55b8812be1b0 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b88169ebb0 con 0x55b8812c6390 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1aa6dd740 10 client.0 _ll_register_callbacks cb 0x55b8815a8990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1a2706640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1a2706640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38022/0 (socket says 172.21.15.28:38022) 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.346+0000 7ff1a2706640 1 -- 172.21.15.28:0/3335833435 learned_addr learned my addr 172.21.15.28:0/3335833435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b88169ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a1704640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b88169bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 -- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 msgr2=0x55b88169bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b88169bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 -- 172.21.15.28:0/3335833435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 msgr2=0x55b88169ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b88169ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 -- 172.21.15.28:0/3335833435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8815bcc40 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1a2706640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff194000c00 tx=0x7ff194004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a736a3464b5abcf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1817fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194029060 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7ff1940050c0 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1940047a0 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.347+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ff1940052c0 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 10 client.17454 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55b8816aa860 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1940047a0 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7ff194024410 con 0x55b8812e7070 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 10 client.17454 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 20 client.17454 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b8812a59f0 0x55b8816adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 4 client.17454 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b881606da0 con 0x55b8812a59f0 2024-09-21T05:55:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.348+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b8812a59f0 0x55b8816adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.349+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b8812a59f0 0x55b8816adc70 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c43c3698095ae9b7 server_cookie=6fc92ddd42a15570 in_seq=0 out_seq=0 2024-09-21T05:55:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.349+0000 7ff1817fa640 10 client.17454 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1817fa640 1 -- 172.21.15.28:0/3335833435 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b881606da0 con 0x55b8812a59f0 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1817fa640 10 client.17454 handle_command_reply: tid=0 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 client.17454 shutdown 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 msgr2=0x55b88169c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff194000c00 tx=0x7ff194004df0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 shutdown_connections 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b8812a59f0 0x55b8816adc70 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812e7070 0x55b88169c4f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b88169bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/3335833435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b88169ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 >> 172.21.15.28:0/3335833435 conn(0x55b8812db460 msgr2=0x55b88168fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1817fa640 0 client.17454 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:42.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 shutdown_connections 2024-09-21T05:55:42.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/3335833435 wait complete. 2024-09-21T05:55:42.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.350+0000 7ff1aa6dd740 20 client.17454 trim_cache size 0 max 16384 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 Processor -- start 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 -- start start 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b881693b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812e7070 0x55b881698c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1ad5c0 con 0x55b8812c6390 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1af1f0 con 0x55b8812be1b0 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1adcf0 con 0x55b8812e7070 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1a1704640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1a1704640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38032/0 (socket says 172.21.15.28:38032) 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.368+0000 7ff1a1704640 1 -- 172.21.15.28:0/2523987921 learned_addr learned my addr 172.21.15.28:0/2523987921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 -- 172.21.15.28:0/2523987921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812e7070 msgr2=0x55b881698c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812e7070 0x55b881698c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b881693b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 -- 172.21.15.28:0/2523987921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 msgr2=0x55b881693b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b881693b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 -- 172.21.15.28:0/2523987921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88169a9d0 con 0x55b8812c6390 2024-09-21T05:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1a1704640 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff1900ca970 tx=0x7ff190031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3865f1e1f83f4e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1837fe640 1 -- 172.21.15.28:0/2523987921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190049070 con 0x55b8812c6390 2024-09-21T05:55:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1837fe640 1 -- 172.21.15.28:0/2523987921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1900cd210 con 0x55b8812c6390 2024-09-21T05:55:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.369+0000 7ff1837fe640 1 -- 172.21.15.28:0/2523987921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190105280 con 0x55b8812c6390 2024-09-21T05:55:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/2523987921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 msgr2=0x55b88169d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff1900ca970 tx=0x7ff190031a10 comp rx=0 tx=0).stop 2024-09-21T05:55:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/2523987921 shutdown_connections 2024-09-21T05:55:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812e7070 0x55b881698c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812c6390 0x55b88169d6a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/2523987921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812be1b0 0x55b881693b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/2523987921 >> 172.21.15.28:0/2523987921 conn(0x55b8812db460 msgr2=0x55b88169cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/2523987921 shutdown_connections 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/2523987921 wait complete. 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 -- start start 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.370+0000 7ff1aa6dd740 1 Processor -- start 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b8816b1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812e7070 0x55b88170d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1aeaf0 con 0x55b8812be1b0 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1ae3f0 con 0x55b8812e7070 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1ab5a0 con 0x55b8812c6390 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1aa6dd740 10 client.0 _ll_register_callbacks cb 0x7ff194014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38042/0 (socket says 172.21.15.28:38042) 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 -- 172.21.15.28:0/4288011835 learned_addr learned my addr 172.21.15.28:0/4288011835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1704640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b8816b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a2706640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812e7070 0x55b88170d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 -- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 msgr2=0x55b8816b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b8816b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 -- 172.21.15.28:0/4288011835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812e7070 msgr2=0x55b88170d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812e7070 0x55b88170d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.371+0000 7ff1a1f05640 1 -- 172.21.15.28:0/4288011835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88170def0 con 0x55b8812be1b0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1a1f05640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff18c004410 tx=0x7ff18c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34320ad4b202d969 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c029840 con 0x55b8812be1b0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7ff18c0299e0 con 0x55b8812be1b0 2024-09-21T05:55:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1900cd210 con 0x55b8812be1b0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ff18c037050 con 0x55b8812be1b0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 10 client.17460 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55b8817185a0 con 0x55b8812be1b0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1900cd210 con 0x55b8812be1b0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7ff18c01a040 con 0x55b8812be1b0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 10 client.17460 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 20 client.17460 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b88127ada0 0x55b88171ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 4 client.17460 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.372+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b881602850 con 0x55b88127ada0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.373+0000 7ff1a1704640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b88127ada0 0x55b88171ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.373+0000 7ff1817fa640 10 client.17460 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.373+0000 7ff1a1704640 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b88127ada0 0x55b88171ada0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2ff63822dc1b27f server_cookie=d762b75f46624d87 in_seq=0 out_seq=0 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.373+0000 7ff1817fa640 1 -- 172.21.15.28:0/4288011835 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3498 (crc 0 0 0) 0x55b881602850 con 0x55b88127ada0 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.373+0000 7ff1817fa640 10 client.17460 handle_command_reply: tid=0 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 client.17460 shutdown 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 msgr2=0x55b8816b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff18c004410 tx=0x7ff18c004a40 comp rx=0 tx=0).stop 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 shutdown_connections 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55b88127ada0 0x55b88171ada0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b8812e7070 0x55b88170d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b8812c6390 0x55b8816b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 --2- 172.21.15.28:0/4288011835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b8812be1b0 0x55b8816b0d90 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1817fa640 0 client.17460 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 >> 172.21.15.28:0/4288011835 conn(0x55b8812db460 msgr2=0x55b8816a6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 shutdown_connections 2024-09-21T05:55:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 1 -- 172.21.15.28:0/4288011835 wait complete. 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.374+0000 7ff1aa6dd740 20 client.17460 trim_cache size 0 max 16384 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:09.151907+0000", 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:55:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:55:42.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 msgr2=0x7ff1840b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 0x7ff1840b8bc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff18c005fd0 tx=0x7ff18c009680 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 msgr2=0x7ff19c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff190032fd0 tx=0x7ff190004270 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 shutdown_connections 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff19c108cf0 0x7ff19c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff1840b66b0 0x7ff1840b8bc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c1083a0 0x7ff19c1999d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 --2- 172.21.15.28:0/3581287903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102a60 0x7ff19c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 >> 172.21.15.28:0/3581287903 conn(0x7ff19c079730 msgr2=0x7ff19c104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 shutdown_connections 2024-09-21T05:55:42.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:42.375+0000 7ff1a4169640 1 -- 172.21.15.28:0/3581287903 wait complete. 2024-09-21T05:55:42.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:42 smithi181 ceph-mon[34819]: pgmap v589: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:42.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:42 smithi028 ceph-mon[31227]: pgmap v589: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:42.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:42 smithi129 ceph-mon[34885]: pgmap v589: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:43.311 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:55:43.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:55:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:44.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 -- 172.21.15.28:0/1985134715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083b0 msgr2=0x7fb4581087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:44.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 --2- 172.21.15.28:0/1985134715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083b0 0x7fb4581087c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fb454009900 tx=0x7fb45402a410 comp rx=0 tx=0).stop 2024-09-21T05:55:44.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 -- 172.21.15.28:0/1985134715 shutdown_connections 2024-09-21T05:55:44.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 --2- 172.21.15.28:0/1985134715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458108d00 0x7fb45810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 --2- 172.21.15.28:0/1985134715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4581083b0 0x7fb4581087c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 --2- 172.21.15.28:0/1985134715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb458102a70 0x7fb458102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 -- 172.21.15.28:0/1985134715 >> 172.21.15.28:0/1985134715 conn(0x7fb458079730 msgr2=0x7fb458079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 -- 172.21.15.28:0/1985134715 shutdown_connections 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.086+0000 7fb46022e640 1 -- 172.21.15.28:0/1985134715 wait complete. 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.087+0000 7fb46022e640 1 Processor -- start 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.087+0000 7fb46022e640 1 -- start start 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458102a70 0x7fb458199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 0x7fb458199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb458112970 con 0x7fb458108d00 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb4581127f0 con 0x7fb458102a70 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb46022e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb458112af0 con 0x7fb4581083b0 2024-09-21T05:55:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38068/0 (socket says 172.21.15.28:38068) 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 -- 172.21.15.28:0/847157666 learned_addr learned my addr 172.21.15.28:0/847157666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45d7c9640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 0x7fb458199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45dfca640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458102a70 0x7fb458199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 -- 172.21.15.28:0/847157666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 msgr2=0x7fb458199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 0x7fb458199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 -- 172.21.15.28:0/847157666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458102a70 msgr2=0x7fb458199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458102a70 0x7fb458199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 -- 172.21.15.28:0/847157666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45819f2d0 con 0x7fb458108d00 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45d7c9640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 0x7fb458199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb45e7cb640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb45000eaa0 tx=0x7fb45000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.088+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45000cdb0 con 0x7fb458108d00 2024-09-21T05:55:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.089+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb450005020 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.089+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45819f5c0 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.089+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4500116d0 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.089+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4581a5d20 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.090+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb458103cf0 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.091+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb4500051d0 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.092+0000 7fb44affd640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 0x7fb4440b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.092+0000 7fb45dfca640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 0x7fb4440b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.092+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fb4500e7cb0 con 0x7fb458108d00 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.093+0000 7fb45dfca640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 0x7fb4440b8b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0058c0 tx=0x7fb44c00a250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.095+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb450016030 con 0x7fb458108d00 2024-09-21T05:55:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.324+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb458102e80 con 0x7fb458108d00 2024-09-21T05:55:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.325+0000 7fb44affd640 1 -- 172.21.15.28:0/847157666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11993 (secure 0 0 0) 0x7fb458102e80 con 0x7fb458108d00 2024-09-21T05:55:44.327 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:55:44.327 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"btime":"2024-09-21T05:55:20:235633+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:55:19.279395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:55:44.328 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 48 2024-09-21T05:55:44.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 msgr2=0x7fb4440b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:44.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 0x7fb4440b8b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0058c0 tx=0x7fb44c00a250 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 msgr2=0x7fb45819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb45000eaa0 tx=0x7fb45000ef60 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 shutdown_connections 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb4440b6680 0x7fb4440b8b90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458108d00 0x7fb45819eba0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4581083b0 0x7fb458199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 --2- 172.21.15.28:0/847157666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb458102a70 0x7fb458199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.327+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 >> 172.21.15.28:0/847157666 conn(0x7fb458079730 msgr2=0x7fb458104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.328+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 shutdown_connections 2024-09-21T05:55:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:44.328+0000 7fb46022e640 1 -- 172.21.15.28:0/847157666 wait complete. 2024-09-21T05:55:44.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:44 smithi181 ceph-mon[34819]: pgmap v590: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:44 smithi028 ceph-mon[31227]: pgmap v590: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:44.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:44 smithi129 ceph-mon[34885]: pgmap v590: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:45.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:55:45.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:45 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/847157666' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:45.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:45 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/847157666' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:45 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/847157666' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:45.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 -- 172.21.15.28:0/2109073139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108cf0 msgr2=0x7fc2a810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2109073139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108cf0 0x7fc2a810f520 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc298009ca0 tx=0x7fc29802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 -- 172.21.15.28:0/2109073139 shutdown_connections 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2109073139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8108cf0 0x7fc2a810f520 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2109073139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 0x7fc2a81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2109073139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8102a60 0x7fc2a8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 -- 172.21.15.28:0/2109073139 >> 172.21.15.28:0/2109073139 conn(0x7fc2a8079730 msgr2=0x7fc2a8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 -- 172.21.15.28:0/2109073139 shutdown_connections 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.827+0000 7fc2ad527640 1 -- 172.21.15.28:0/2109073139 wait complete. 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 Processor -- start 2024-09-21T05:55:45.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 -- start start 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 0x7fc2a8199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8108cf0 0x7fc2a8199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8112960 con 0x7fc2a8102a60 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81127e0 con 0x7fc2a8108cf0 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2ad527640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8112ae0 con 0x7fc2a81083a0 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46012/0 (socket says 172.21.15.28:46012) 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.828+0000 7fc2a6ffd640 1 -- 172.21.15.28:0/2224516707 learned_addr learned my addr 172.21.15.28:0/2224516707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8108cf0 0x7fc2a8199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 0x7fc2a8199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 -- 172.21.15.28:0/2224516707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 msgr2=0x7fc2a8199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 0x7fc2a8199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 -- 172.21.15.28:0/2224516707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8108cf0 msgr2=0x7fc2a8199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8108cf0 0x7fc2a8199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 -- 172.21.15.28:0/2224516707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a81a8680 con 0x7fc2a8102a60 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000d950 tx=0x7fc2a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc29ffff640 1 -- 172.21.15.28:0/2224516707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0017070 con 0x7fc2a8102a60 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc29ffff640 1 -- 172.21.15.28:0/2224516707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0009c30 con 0x7fc2a8102a60 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a81a8910 con 0x7fc2a8102a60 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc29ffff640 1 -- 172.21.15.28:0/2224516707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000f620 con 0x7fc2a8102a60 2024-09-21T05:55:45.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.829+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a81a8ea0 con 0x7fc2a8102a60 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 Processor -- start 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 -- start start 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc29ffff640 1 -- 172.21.15.28:0/2224516707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc2a0010460 con 0x7fc2a8102a60 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e11b0 0x5635c26b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e9390 0x5635c26b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635c2627ea0 con 0x5635c230a070 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5635c26172d0 con 0x5635c22e11b0 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5635c261b0e0 con 0x5635c22e9390 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46022/0 (socket says 172.21.15.28:46022) 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.832+0000 7fc2a77fe640 1 -- 172.21.15.28:0/1783239948 learned_addr learned my addr 172.21.15.28:0/1783239948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e11b0 0x5635c26b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e9390 0x5635c26b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 -- 172.21.15.28:0/1783239948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e9390 msgr2=0x5635c26b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e9390 0x5635c26b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 -- 172.21.15.28:0/1783239948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e11b0 msgr2=0x5635c26b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e11b0 0x5635c26b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 -- 172.21.15.28:0/1783239948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635c2375650 con 0x5635c230a070 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc298009930 tx=0x7fc298009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509fd0de9615951d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc29ffff640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 0x7fc2940b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 0x7fc2940b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.833+0000 7fc29d7fa640 1 -- 172.21.15.28:0/1783239948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298031df0 con 0x5635c230a070 2024-09-21T05:55:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc29d7fa640 1 -- 172.21.15.28:0/1783239948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298036040 con 0x5635c230a070 2024-09-21T05:55:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc29d7fa640 1 -- 172.21.15.28:0/1783239948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2980042a0 con 0x5635c230a070 2024-09-21T05:55:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc29ffff640 1 -- 172.21.15.28:0/2224516707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc2a00e73b0 con 0x7fc2a8102a60 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/1783239948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 msgr2=0x5635c26b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2a6ffd640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 0x7fc2940b8c10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00eafd0 tx=0x7fc2a00eaea0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc298009930 tx=0x7fc298009960 comp rx=0 tx=0).stop 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/1783239948 shutdown_connections 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e9390 0x5635c26b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e11b0 0x5635c26b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/1783239948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26b12c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.834+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/1783239948 >> 172.21.15.28:0/1783239948 conn(0x5635c22fe460 msgr2=0x5635c26af1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/1783239948 shutdown_connections 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/1783239948 wait complete. 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- start start 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 Processor -- start 2024-09-21T05:55:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26bca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26bcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635c2478700 con 0x5635c230a070 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5635c25b2160 con 0x5635c22e9390 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5635c26bfa40 con 0x5635c22e11b0 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.835+0000 7fc2b3a9b740 10 client.0 _ll_register_callbacks cb 0x5635c25cb9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46036/0 (socket says 172.21.15.28:46036) 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 -- 172.21.15.28:0/317689621 learned_addr learned my addr 172.21.15.28:0/317689621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26bcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 -- 172.21.15.28:0/317689621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 msgr2=0x5635c26bca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26bca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 -- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 msgr2=0x5635c26bcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26bcf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 -- 172.21.15.28:0/317689621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635c25dfc90 con 0x5635c230a070 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc298003d10 tx=0x7fc29803e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda37c8559fe14fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc28e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298059060 con 0x5635c230a070 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc298031650 con 0x5635c230a070 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.836+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc298036040 con 0x5635c230a070 2024-09-21T05:55:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc298031e40 con 0x5635c230a070 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 10 client.17478 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5635c26cb790 con 0x5635c230a070 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298036040 con 0x5635c230a070 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fc298004580 con 0x5635c230a070 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 10 client.17478 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 10 client.17478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 20 client.17478 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c22c89f0 0x5635c26ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 4 client.17478 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635c2629df0 con 0x5635c22c89f0 2024-09-21T05:55:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.837+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c22c89f0 0x5635c26ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.838+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c22c89f0 0x5635c26ceaf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4cd8f7f4bd9b1f99 server_cookie=f67c2e734bae56db in_seq=0 out_seq=0 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.838+0000 7fc28e7fc640 10 client.17478 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.838+0000 7fc28e7fc640 1 -- 172.21.15.28:0/317689621 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5635c2629df0 con 0x5635c22c89f0 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.838+0000 7fc28e7fc640 10 client.17478 handle_command_reply: tid=0 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 client.17478 shutdown 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 msgr2=0x5635c26c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc298003d10 tx=0x7fc29803e040 comp rx=0 tx=0).stop 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 shutdown_connections 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c22c89f0 0x5635c26ceaf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc28e7fc640 0 client.17478 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c230a070 0x5635c26c8d30 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26bcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/317689621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26bca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 >> 172.21.15.28:0/317689621 conn(0x5635c22fe460 msgr2=0x5635c26b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 shutdown_connections 2024-09-21T05:55:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/317689621 wait complete. 2024-09-21T05:55:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.839+0000 7fc2b3a9b740 20 client.17478 trim_cache size 0 max 16384 2024-09-21T05:55:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.852+0000 7fc2b3a9b740 1 Processor -- start 2024-09-21T05:55:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.852+0000 7fc2b3a9b740 1 -- start start 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26ba150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81b19b0 con 0x5635c22c89f0 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81b35e0 con 0x5635c22e9390 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81b20e0 con 0x5635c22e11b0 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46048/0 (socket says 172.21.15.28:46048) 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 -- 172.21.15.28:0/3507431559 learned_addr learned my addr 172.21.15.28:0/3507431559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26ba150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 -- 172.21.15.28:0/3507431559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 msgr2=0x5635c26b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26b9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 -- 172.21.15.28:0/3507431559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 msgr2=0x5635c26ba150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26ba150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.853+0000 7fc2a77fe640 1 -- 172.21.15.28:0/3507431559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635c26bf5d0 con 0x5635c22c89f0 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.854+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc29802ade0 tx=0x7fc298009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331227ad0d8db6b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.854+0000 7fc29cff9640 1 -- 172.21.15.28:0/3507431559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298059070 con 0x5635c22c89f0 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.854+0000 7fc29cff9640 1 -- 172.21.15.28:0/3507431559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298036070 con 0x5635c22c89f0 2024-09-21T05:55:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.854+0000 7fc29cff9640 1 -- 172.21.15.28:0/3507431559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29802fc00 con 0x5635c22c89f0 2024-09-21T05:55:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/3507431559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 msgr2=0x5635c26b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc29802ade0 tx=0x7fc298009930 comp rx=0 tx=0).stop 2024-09-21T05:55:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/3507431559 shutdown_connections 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26ba150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/3507431559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26b4ad0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/3507431559 >> 172.21.15.28:0/3507431559 conn(0x5635c22fe460 msgr2=0x5635c26b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/3507431559 shutdown_connections 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/3507431559 wait complete. 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 -- start start 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.855+0000 7fc2b3a9b740 1 Processor -- start 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81b2ee0 con 0x5635c22c89f0 2024-09-21T05:55:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81b27e0 con 0x5635c22e9390 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81af990 con 0x5635c22e11b0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2b3a9b740 10 client.0 _ll_register_callbacks cb 0x7fc2a00ead70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46060/0 (socket says 172.21.15.28:46060) 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 -- 172.21.15.28:0/2036687948 learned_addr learned my addr 172.21.15.28:0/2036687948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 -- 172.21.15.28:0/2036687948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 msgr2=0x5635c26cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 -- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 msgr2=0x5635c26cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 -- 172.21.15.28:0/2036687948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635c26d0110 con 0x5635c22c89f0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc2a77fe640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc298003a50 tx=0x7fc298009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3be0bf97529dbc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.856+0000 7fc28e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298069060 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc298036070 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc29802fe10 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc298059040 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 10 client.17484 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5635c2738fe0 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298036070 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fc298030650 con 0x5635c22c89f0 2024-09-21T05:55:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 10 client.17484 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:45.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 10 client.17484 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:45.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 20 client.17484 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:45.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c230a070 0x5635c273b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:45.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 4 client.17484 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:45.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.857+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635c26258a0 con 0x5635c230a070 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.858+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c230a070 0x5635c273b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2a67fc640 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c230a070 0x5635c273b780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=788ebd9fe44d7428 server_cookie=6483da2e06c44aa5 in_seq=0 out_seq=0 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc28e7fc640 10 client.17484 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc28e7fc640 1 -- 172.21.15.28:0/2036687948 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635c26258a0 con 0x5635c230a070 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc28e7fc640 10 client.17484 handle_command_reply: tid=0 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 client.17484 shutdown 2024-09-21T05:55:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 msgr2=0x5635c26d2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc298003a50 tx=0x7fc298009b90 comp rx=0 tx=0).stop 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 shutdown_connections 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x5635c230a070 0x5635c273b780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5635c22e9390 0x5635c26cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635c22e11b0 0x5635c26cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 --2- 172.21.15.28:0/2036687948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635c22c89f0 0x5635c26d2140 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 >> 172.21.15.28:0/2036687948 conn(0x5635c22fe460 msgr2=0x5635c26bc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc28e7fc640 0 client.17484 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 shutdown_connections 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 1 -- 172.21.15.28:0/2036687948 wait complete. 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.859+0000 7fc2b3a9b740 20 client.17484 trim_cache size 0 max 16384 2024-09-21T05:55:45.865 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 msgr2=0x7fc2940b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 0x7fc2940b8c10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00eafd0 tx=0x7fc2a00eaea0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 msgr2=0x7fc2a8199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000d950 tx=0x7fc2a000de10 comp rx=0 tx=0).stop 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 shutdown_connections 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a8108cf0 0x7fc2a8199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2a81083a0 0x7fc2a8199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc2940b6700 0x7fc2940b8c10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 --2- 172.21.15.28:0/2224516707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2a8102a60 0x7fc2a8199000 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 >> 172.21.15.28:0/2224516707 conn(0x7fc2a8079730 msgr2=0x7fc2a81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 shutdown_connections 2024-09-21T05:55:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:45.860+0000 7fc2ad527640 1 -- 172.21.15.28:0/2224516707 wait complete. 2024-09-21T05:55:46.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:46 smithi028 ceph-mon[31227]: pgmap v591: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:46.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:46 smithi129 ceph-mon[34885]: pgmap v591: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:46.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:55:46.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:46 smithi181 ceph-mon[34819]: pgmap v591: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 -- 172.21.15.28:0/2988829753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 msgr2=0x7f9f04103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 --2- 172.21.15.28:0/2988829753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f04103680 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009900 tx=0x7f9ef402a410 comp rx=0 tx=0).stop 2024-09-21T05:55:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 -- 172.21.15.28:0/2988829753 shutdown_connections 2024-09-21T05:55:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 --2- 172.21.15.28:0/2988829753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105b90 0x7f9f0410f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 --2- 172.21.15.28:0/2988829753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105240 0x7f9f04105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 --2- 172.21.15.28:0/2988829753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f04103680 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 -- 172.21.15.28:0/2988829753 >> 172.21.15.28:0/2988829753 conn(0x7f9f04079ec0 msgr2=0x7f9f0407a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 -- 172.21.15.28:0/2988829753 shutdown_connections 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.344+0000 7f9f094a6640 1 -- 172.21.15.28:0/2988829753 wait complete. 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.345+0000 7f9f094a6640 1 Processor -- start 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.345+0000 7f9f094a6640 1 -- start start 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105240 0x7f9f0419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 0x7f9f0419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04112a30 con 0x7f9f04103270 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041128b0 con 0x7f9f04105b90 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f094a6640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04112bb0 con 0x7f9f04105240 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46072/0 (socket says 172.21.15.28:46072) 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 -- 172.21.15.28:0/3456584379 learned_addr learned my addr 172.21.15.28:0/3456584379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105240 0x7f9f0419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 0x7f9f0419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 -- 172.21.15.28:0/3456584379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105240 msgr2=0x7f9f0419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105240 0x7f9f0419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 -- 172.21.15.28:0/3456584379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 msgr2=0x7f9f0419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 0x7f9f0419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 -- 172.21.15.28:0/3456584379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f041a65a0 con 0x7f9f04103270 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 0x7f9f0419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.346+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005940 tx=0x7f9ef40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.347+0000 7f9f00ff9640 1 -- 172.21.15.28:0/3456584379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef403f030 con 0x7f9f04103270 2024-09-21T05:55:47.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.347+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f041a6830 con 0x7f9f04103270 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.347+0000 7f9f00ff9640 1 -- 172.21.15.28:0/3456584379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ef4003cb0 con 0x7f9f04103270 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.347+0000 7f9f00ff9640 1 -- 172.21.15.28:0/3456584379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4035070 con 0x7f9f04103270 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.347+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f041a6d10 con 0x7f9f04103270 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.349+0000 7f9f0fa1a740 1 Processor -- start 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.349+0000 7f9f0fa1a740 1 -- start start 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f00ff9640 1 -- 172.21.15.28:0/3456584379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9ef4042030 con 0x7f9f04103270 2024-09-21T05:55:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f56c070 0x56442f9136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f5431b0 0x56442f913be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56442f889ea0 con 0x56442f54b390 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56442f8792d0 con 0x56442f56c070 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56442f87d0e0 con 0x56442f5431b0 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46082/0 (socket says 172.21.15.28:46082) 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f56c070 0x56442f9136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 -- 172.21.15.28:0/1110052290 learned_addr learned my addr 172.21.15.28:0/1110052290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f5431b0 0x56442f913be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 -- 172.21.15.28:0/1110052290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f5431b0 msgr2=0x56442f913be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f5431b0 0x56442f913be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 -- 172.21.15.28:0/1110052290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f56c070 msgr2=0x56442f9136a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f56c070 0x56442f9136a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.350+0000 7f9efa7fc640 1 -- 172.21.15.28:0/1110052290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56442f5d7650 con 0x56442f54b390 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00bd00 tx=0x7f9eec0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de45eddc090cf8d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9f00ff9640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 0x7f9ef00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9ef97fa640 1 -- 172.21.15.28:0/1110052290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec01c680 con 0x56442f54b390 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9ef97fa640 1 -- 172.21.15.28:0/1110052290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eec01ccd0 con 0x56442f54b390 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9ef97fa640 1 -- 172.21.15.28:0/1110052290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec011bf0 con 0x56442f54b390 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 0x7f9ef00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.351+0000 7f9f00ff9640 1 -- 172.21.15.28:0/3456584379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f9ef4107fd0 con 0x7f9f04103270 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1110052290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 msgr2=0x56442f914120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 0x7f9ef00b8b90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9efc005fd0 tx=0x7f9efc009780 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00bd00 tx=0x7f9eec0107a0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1110052290 shutdown_connections 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f914120 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f5431b0 0x56442f913be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1110052290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f56c070 0x56442f9136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1110052290 >> 172.21.15.28:0/1110052290 conn(0x56442f560460 msgr2=0x56442f911590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1110052290 shutdown_connections 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1110052290 wait complete. 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.352+0000 7f9f0fa1a740 1 -- start start 2024-09-21T05:55:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 Processor -- start 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f54b390 0x56442f9235d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f56c070 0x56442f923b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56442f6da700 con 0x56442f5431b0 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56442f814160 con 0x56442f54b390 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56442f9261b0 con 0x56442f56c070 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f0fa1a740 10 client.0 _ll_register_callbacks cb 0x56442f82d9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46090/0 (socket says 172.21.15.28:46090) 2024-09-21T05:55:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 -- 172.21.15.28:0/1562452873 learned_addr learned my addr 172.21.15.28:0/1562452873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f037fe640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f56c070 0x56442f923b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 -- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f56c070 msgr2=0x56442f923b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f56c070 0x56442f923b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 -- 172.21.15.28:0/1562452873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f54b390 msgr2=0x56442f9235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f54b390 0x56442f9235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f54b390 0x56442f9235d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.353+0000 7f9f02ffd640 1 -- 172.21.15.28:0/1562452873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56442f841c90 con 0x56442f5431b0 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40cdb20 tx=0x7f9ef403b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29d1cc68727afe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:47.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4049070 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f9ef40b50b0 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eec01ccd0 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9ef40cd770 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 10 client.17496 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56442f931e80 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eec01ccd0 con 0x56442f5431b0 2024-09-21T05:55:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f9ef41059a0 con 0x56442f5431b0 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 10 client.17496 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 10 client.17496 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.354+0000 7f9f0fa1a740 20 client.17496 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f52a9f0 0x56442f935340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9f0fa1a740 4 client.17496 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56442f88bdf0 con 0x56442f52a9f0 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f52a9f0 0x56442f935340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9eea7fc640 10 client.17496 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.355+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f52a9f0 0x56442f935340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c7eeafeeef0985a8 server_cookie=b0e7f2b9d4441a6b in_seq=0 out_seq=0 2024-09-21T05:55:47.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9eea7fc640 1 -- 172.21.15.28:0/1562452873 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56442f88bdf0 con 0x56442f52a9f0 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9eea7fc640 10 client.17496 handle_command_reply: tid=0 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 client.17496 shutdown 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 msgr2=0x56442f923090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40cdb20 tx=0x7f9ef403b6b0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 shutdown_connections 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f52a9f0 0x56442f935340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f56c070 0x56442f923b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9eea7fc640 0 client.17496 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f54b390 0x56442f9235d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/1562452873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f5431b0 0x56442f923090 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.356+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 >> 172.21.15.28:0/1562452873 conn(0x56442f560460 msgr2=0x56442f916830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.357+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 shutdown_connections 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.357+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/1562452873 wait complete. 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.357+0000 7f9f0fa1a740 20 client.17496 trim_cache size 0 max 16384 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 Processor -- start 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 -- start start 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f924f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f914de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041ad550 con 0x56442f54b390 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041af180 con 0x56442f5431b0 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.370+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041adc80 con 0x56442f52a9f0 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46104/0 (socket says 172.21.15.28:46104) 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 -- 172.21.15.28:0/2581596709 learned_addr learned my addr 172.21.15.28:0/2581596709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f914de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f924f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 -- 172.21.15.28:0/2581596709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 msgr2=0x56442f924f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f924f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 -- 172.21.15.28:0/2581596709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 msgr2=0x56442f914de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f914de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 -- 172.21.15.28:0/2581596709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56442f926090 con 0x56442f54b390 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f914de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9f037fe640 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009db0 tx=0x7f9efc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525fff72ef790b56 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.371+0000 7f9ef8ff9640 1 -- 172.21.15.28:0/2581596709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc004d20 con 0x56442f54b390 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.372+0000 7f9ef8ff9640 1 -- 172.21.15.28:0/2581596709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9efc020a80 con 0x56442f54b390 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.372+0000 7f9ef8ff9640 1 -- 172.21.15.28:0/2581596709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc005720 con 0x56442f54b390 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/2581596709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 msgr2=0x56442f910bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009db0 tx=0x7f9efc016000 comp rx=0 tx=0).stop 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/2581596709 shutdown_connections 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f910bc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f914de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/2581596709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f924f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/2581596709 >> 172.21.15.28:0/2581596709 conn(0x56442f560460 msgr2=0x56442f9146d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/2581596709 shutdown_connections 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/2581596709 wait complete. 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 -- start start 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.373+0000 7f9f0fa1a740 1 Processor -- start 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f927de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f928380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041aea80 con 0x56442f54b390 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041ae380 con 0x56442f5431b0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041ab530 con 0x56442f52a9f0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f0fa1a740 10 client.0 _ll_register_callbacks cb 0x7f9ef4005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46110/0 (socket says 172.21.15.28:46110) 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 -- 172.21.15.28:0/3372347747 learned_addr learned my addr 172.21.15.28:0/3372347747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9efa7fc640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f928380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f927de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 -- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 msgr2=0x56442f927de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f927de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 -- 172.21.15.28:0/3372347747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 msgr2=0x56442f928380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f928380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 -- 172.21.15.28:0/3372347747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56442f9993b0 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9f037fe640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009d80 tx=0x7f9efc020d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5509e90aa748e71 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9eea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.374+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc03a060 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f9efc0207e0 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9efc020a80 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9efc004a40 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 10 client.17502 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56442f9a06d0 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9efc020a80 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f9efc00f040 con 0x56442f54b390 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 10 client.17502 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 10 client.17502 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:55:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 20 client.17502 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f56c070 0x56442f9a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 4 client.17502 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.375+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56442f8878a0 con 0x56442f56c070 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f56c070 0x56442f9a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9eea7fc640 10 client.17502 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f02ffd640 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f56c070 0x56442f9a2d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c3ac72bdce937c7a server_cookie=b0ed05d66e83362d in_seq=0 out_seq=0 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9eea7fc640 1 -- 172.21.15.28:0/3372347747 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56442f8878a0 con 0x56442f56c070 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9eea7fc640 10 client.17502 handle_command_reply: tid=0 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 client.17502 shutdown 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 msgr2=0x56442f998d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009d80 tx=0x7f9efc020d90 comp rx=0 tx=0).stop 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 shutdown_connections 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56442f56c070 0x56442f9a2d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56442f54b390 0x56442f998d40 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56442f5431b0 0x56442f928380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 --2- 172.21.15.28:0/3372347747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56442f52a9f0 0x56442f927de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9eea7fc640 0 client.17502 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.376+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 >> 172.21.15.28:0/3372347747 conn(0x56442f560460 msgr2=0x56442f924bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 shutdown_connections 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f0fa1a740 1 -- 172.21.15.28:0/3372347747 wait complete. 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f0fa1a740 20 client.17502 trim_cache size 0 max 16384 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 msgr2=0x7f9ef00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 0x7f9ef00b8b90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9efc005fd0 tx=0x7f9efc009780 comp rx=0 tx=0).stop 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 msgr2=0x7f9f0419b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005940 tx=0x7f9ef40042f0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 shutdown_connections 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f04105b90 0x7f9f0419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f04105240 0x7f9f0419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9ef00b6680 0x7f9ef00b8b90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 --2- 172.21.15.28:0/3456584379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f04103270 0x7f9f0419b2e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 >> 172.21.15.28:0/3456584379 conn(0x7f9f04079ec0 msgr2=0x7f9f041073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.377+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 shutdown_connections 2024-09-21T05:55:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:47.378+0000 7f9f094a6640 1 -- 172.21.15.28:0/3456584379 wait complete. 2024-09-21T05:55:48.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:55:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:48 smithi028 ceph-mon[31227]: pgmap v592: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:48.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:48 smithi129 ceph-mon[34885]: pgmap v592: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:48.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:48 smithi181 ceph-mon[34819]: pgmap v592: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:55:49.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 -- 172.21.15.28:0/129006065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964104a30 msgr2=0x7fd964104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/129006065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964104a30 0x7fd964104e40 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd954009900 tx=0x7fd95402a410 comp rx=0 tx=0).stop 2024-09-21T05:55:49.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 -- 172.21.15.28:0/129006065 shutdown_connections 2024-09-21T05:55:49.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/129006065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd964110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/129006065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964105380 0x7fd96410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/129006065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964104a30 0x7fd964104e40 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 -- 172.21.15.28:0/129006065 >> 172.21.15.28:0/129006065 conn(0x7fd964079760 msgr2=0x7fd964079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 -- 172.21.15.28:0/129006065 shutdown_connections 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 -- 172.21.15.28:0/129006065 wait complete. 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.994+0000 7fd96a9f2640 1 Processor -- start 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 -- start start 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964104a30 0x7fd964102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd9641031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd964113ac0 con 0x7fd964105380 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd964113940 con 0x7fd96410de60 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd96a9f2640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd964113c40 con 0x7fd964104a30 2024-09-21T05:55:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46126/0 (socket says 172.21.15.28:46126) 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 -- 172.21.15.28:0/3765366010 learned_addr learned my addr 172.21.15.28:0/3765366010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd963fff640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964104a30 0x7fd964102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd9641031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 -- 172.21.15.28:0/3765366010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964104a30 msgr2=0x7fd964102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964104a30 0x7fd964102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 -- 172.21.15.28:0/3765366010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 msgr2=0x7fd9641031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd9641031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd9637fe640 1 -- 172.21.15.28:0/3765366010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9641a8720 con 0x7fd964105380 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.995+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd9641031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c970 tx=0x7fd95800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd9617fa640 1 -- 172.21.15.28:0/3765366010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958004220 con 0x7fd964105380 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd9617fa640 1 -- 172.21.15.28:0/3765366010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9580043c0 con 0x7fd964105380 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9641a8a10 con 0x7fd964105380 2024-09-21T05:55:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd9617fa640 1 -- 172.21.15.28:0/3765366010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958004a00 con 0x7fd964105380 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.996+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641a8f00 con 0x7fd964105380 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.999+0000 7fd970f66740 1 Processor -- start 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:48.999+0000 7fd970f66740 1 -- start start 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc41581b0 0x55edc4528800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4528d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55edc449eea0 con 0x55edc4181070 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55edc448e2d0 con 0x55edc41581b0 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd970f66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55edc44920e0 con 0x55edc4160390 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd9617fa640 1 -- 172.21.15.28:0/3765366010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fd9580079b0 con 0x7fd964105380 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46140/0 (socket says 172.21.15.28:46140) 2024-09-21T05:55:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 -- 172.21.15.28:0/3080279479 learned_addr learned my addr 172.21.15.28:0/3080279479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4528d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd963fff640 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc41581b0 0x55edc4528800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 -- 172.21.15.28:0/3080279479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 msgr2=0x55edc4528d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4528d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 -- 172.21.15.28:0/3080279479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc41581b0 msgr2=0x55edc4528800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc41581b0 0x55edc4528800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 -- 172.21.15.28:0/3080279479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edc41ec650 con 0x55edc4181070 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.000+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd95c008e60 tx=0x7fd95c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769c58254a4e6d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd94a7fc640 1 -- 172.21.15.28:0/3080279479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00f830 con 0x55edc4181070 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd94a7fc640 1 -- 172.21.15.28:0/3080279479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd95c018480 con 0x55edc4181070 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd94a7fc640 1 -- 172.21.15.28:0/3080279479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00e610 con 0x55edc4181070 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd9617fa640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 0x7fd94c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd963fff640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 0x7fd94c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.001+0000 7fd9617fa640 1 -- 172.21.15.28:0/3765366010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fd9580e7580 con 0x7fd964105380 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- 172.21.15.28:0/3080279479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 msgr2=0x55edc45282c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd963fff640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 0x7fd94c0b8b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd954009230 tx=0x7fd954031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd95c008e60 tx=0x7fd95c00d710 comp rx=0 tx=0).stop 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- 172.21.15.28:0/3080279479 shutdown_connections 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4528d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc41581b0 0x55edc4528800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 --2- 172.21.15.28:0/3080279479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc4181070 0x55edc45282c0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- 172.21.15.28:0/3080279479 >> 172.21.15.28:0/3080279479 conn(0x55edc4175460 msgr2=0x55edc45261b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- 172.21.15.28:0/3080279479 shutdown_connections 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- 172.21.15.28:0/3080279479 wait complete. 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.002+0000 7fd970f66740 1 -- start start 2024-09-21T05:55:49.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 Processor -- start 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4533f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4181070 0x55edc453fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55edc42ef700 con 0x55edc41581b0 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55edc4429160 con 0x55edc4181070 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55edc4536aa0 con 0x55edc4160390 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd970f66740 10 client.0 _ll_register_callbacks cb 0x55edc44429e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46146/0 (socket says 172.21.15.28:46146) 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 -- 172.21.15.28:0/3965251799 learned_addr learned my addr 172.21.15.28:0/3965251799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4533f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd968f8f640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4181070 0x55edc453fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 -- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 msgr2=0x55edc4533f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4533f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 -- 172.21.15.28:0/3965251799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4181070 msgr2=0x55edc453fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4181070 0x55edc453fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.003+0000 7fd963fff640 1 -- 172.21.15.28:0/3965251799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edc4456c90 con 0x55edc41581b0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd963fff640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd96406c8f0 tx=0x7fd954009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae98ddbab1db493 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95404c070 con 0x55edc41581b0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fd9540037f0 con 0x55edc41581b0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c018480 con 0x55edc41581b0 2024-09-21T05:55:49.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd954058050 con 0x55edc41581b0 2024-09-21T05:55:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd970f66740 10 client.17514 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55edc4542830 con 0x55edc41581b0 2024-09-21T05:55:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd95c018480 con 0x55edc41581b0 2024-09-21T05:55:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.004+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fd954048b00 con 0x55edc41581b0 2024-09-21T05:55:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 10 client.17514 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 10 client.17514 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 20 client.17514 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc413f9f0 0x55edc4545dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 4 client.17514 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edc44a0df0 con 0x55edc413f9f0 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc413f9f0 0x55edc4545dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd9437fe640 10 client.17514 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.005+0000 7fd9637fe640 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc413f9f0 0x55edc4545dc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be1759cfe5b8364d server_cookie=32d6d1571f38bf98 in_seq=0 out_seq=0 2024-09-21T05:55:49.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.006+0000 7fd9437fe640 1 -- 172.21.15.28:0/3965251799 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55edc44a0df0 con 0x55edc413f9f0 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.006+0000 7fd9437fe640 10 client.17514 handle_command_reply: tid=0 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.006+0000 7fd970f66740 1 client.17514 shutdown 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 msgr2=0x55edc4533a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd96406c8f0 tx=0x7fd954009260 comp rx=0 tx=0).stop 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 shutdown_connections 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc413f9f0 0x55edc4545dc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4181070 0x55edc453fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc4533f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd9437fe640 0 client.17514 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 --2- 172.21.15.28:0/3965251799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc4533a20 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 >> 172.21.15.28:0/3965251799 conn(0x55edc4175460 msgr2=0x55edc452b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 shutdown_connections 2024-09-21T05:55:49.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 1 -- 172.21.15.28:0/3965251799 wait complete. 2024-09-21T05:55:49.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.007+0000 7fd970f66740 20 client.17514 trim_cache size 0 max 16384 2024-09-21T05:55:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.020+0000 7fd970f66740 1 Processor -- start 2024-09-21T05:55:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.020+0000 7fd970f66740 1 -- start start 2024-09-21T05:55:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc413f9f0 0x55edc4526200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc452b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b1950 con 0x55edc41581b0 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b3580 con 0x55edc413f9f0 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd970f66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b2080 con 0x55edc4160390 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46158/0 (socket says 172.21.15.28:46158) 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 -- 172.21.15.28:0/215964415 learned_addr learned my addr 172.21.15.28:0/215964415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd963fff640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc413f9f0 0x55edc4526200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd968f8f640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc452b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 -- 172.21.15.28:0/215964415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 msgr2=0x55edc452b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc452b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 -- 172.21.15.28:0/215964415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc413f9f0 msgr2=0x55edc4526200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc413f9f0 0x55edc4526200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 -- 172.21.15.28:0/215964415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edc4534ce0 con 0x55edc41581b0 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd968f8f640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc452b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd9637fe640 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c940 tx=0x7fd95800af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9953ff42cde97b30 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.021+0000 7fd949ffb640 1 -- 172.21.15.28:0/215964415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95801d070 con 0x55edc41581b0 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd949ffb640 1 -- 172.21.15.28:0/215964415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9580056a0 con 0x55edc41581b0 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd949ffb640 1 -- 172.21.15.28:0/215964415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9580e4f50 con 0x55edc41581b0 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 -- 172.21.15.28:0/215964415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 msgr2=0x55edc453fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c940 tx=0x7fd95800af60 comp rx=0 tx=0).stop 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 -- 172.21.15.28:0/215964415 shutdown_connections 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc4160390 0x55edc452b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc41581b0 0x55edc453fc90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 --2- 172.21.15.28:0/215964415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc413f9f0 0x55edc4526200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 -- 172.21.15.28:0/215964415 >> 172.21.15.28:0/215964415 conn(0x55edc4175460 msgr2=0x55edc4525af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.022+0000 7fd970f66740 1 -- 172.21.15.28:0/215964415 shutdown_connections 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 -- 172.21.15.28:0/215964415 wait complete. 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 -- start start 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 Processor -- start 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 0x55edc4537f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc41581b0 0x55edc45384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 0x55edc45af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b2e80 con 0x55edc413f9f0 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b2780 con 0x55edc4160390 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.023+0000 7fd970f66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641af930 con 0x55edc41581b0 2024-09-21T05:55:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd970f66740 10 client.0 _ll_register_callbacks cb 0x7fd954009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd963fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 0x55edc4537f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd963fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 0x55edc4537f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46168/0 (socket says 172.21.15.28:46168) 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd963fff640 1 -- 172.21.15.28:0/4010482031 learned_addr learned my addr 172.21.15.28:0/4010482031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 0x55edc45af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd9637fe640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc41581b0 0x55edc45384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 -- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc41581b0 msgr2=0x55edc45384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc41581b0 0x55edc45384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 -- 172.21.15.28:0/4010482031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 msgr2=0x55edc4537f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 0x55edc4537f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 -- 172.21.15.28:0/4010482031 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edc4538cb0 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd968f8f640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 0x55edc45af780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55edc4536640 tx=0x7fd95c008ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dee47a44589fda6c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.024+0000 7fd9437fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00fa40 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9580056a0 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fd95c00fbe0 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd95c031050 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 10 client.27035 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55edc45b0c60 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9580056a0 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fd95c018480 con 0x55edc4160390 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 10 client.27035 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 10 client.27035 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:55:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 20 client.27035 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc4181070 0x55edc45b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 4 client.27035 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.025+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55edc449c8a0 con 0x55edc4181070 2024-09-21T05:55:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd963fff640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc4181070 0x55edc45b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd963fff640 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc4181070 0x55edc45b32b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e019ef998d1e86e1 server_cookie=7910a45d02ecde4f in_seq=0 out_seq=0 2024-09-21T05:55:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd9437fe640 10 client.27035 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd9437fe640 1 -- 172.21.15.28:0/4010482031 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55edc449c8a0 con 0x55edc4181070 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd9437fe640 10 client.27035 handle_command_reply: tid=0 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.026+0000 7fd970f66740 1 client.27035 shutdown 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 msgr2=0x55edc45af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 0x55edc45af780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55edc4536640 tx=0x7fd95c008ba0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 shutdown_connections 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55edc4181070 0x55edc45b32b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edc4160390 0x55edc45af780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55edc41581b0 0x55edc45384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 --2- 172.21.15.28:0/4010482031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55edc413f9f0 0x55edc4537f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd9437fe640 0 client.27035 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:55:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 >> 172.21.15.28:0/4010482031 conn(0x55edc4175460 msgr2=0x55edc452b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 shutdown_connections 2024-09-21T05:55:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 1 -- 172.21.15.28:0/4010482031 wait complete. 2024-09-21T05:55:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd970f66740 20 client.27035 trim_cache size 0 max 16384 2024-09-21T05:55:49.038 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 msgr2=0x7fd94c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 0x7fd94c0b8b90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd954009230 tx=0x7fd954031400 comp rx=0 tx=0).stop 2024-09-21T05:55:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 msgr2=0x7fd964102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.027+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd95800c970 tx=0x7fd95800ce30 comp rx=0 tx=0).stop 2024-09-21T05:55:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 shutdown_connections 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96410de60 0x7fd9641031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fd94c0b6680 0x7fd94c0b8b90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd964105380 0x7fd964102cb0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 --2- 172.21.15.28:0/3765366010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd964104a30 0x7fd964102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 >> 172.21.15.28:0/3765366010 conn(0x7fd964079760 msgr2=0x7fd9641008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 shutdown_connections 2024-09-21T05:55:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:49.028+0000 7fd96a9f2640 1 -- 172.21.15.28:0/3765366010 wait complete. 2024-09-21T05:55:49.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:55:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:50.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:50 smithi028 ceph-mon[31227]: pgmap v593: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 -- 172.21.15.28:0/563175104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140109e40 msgr2=0x7fc140110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 --2- 172.21.15.28:0/563175104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140109e40 0x7fc140110670 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc138009900 tx=0x7fc13802a410 comp rx=0 tx=0).stop 2024-09-21T05:55:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 -- 172.21.15.28:0/563175104 shutdown_connections 2024-09-21T05:55:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 --2- 172.21.15.28:0/563175104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140109e40 0x7fc140110670 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 --2- 172.21.15.28:0/563175104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1401094f0 0x7fc140109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 --2- 172.21.15.28:0/563175104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc140103bb0 0x7fc140103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 -- 172.21.15.28:0/563175104 >> 172.21.15.28:0/563175104 conn(0x7fc1400ff600 msgr2=0x7fc140101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 -- 172.21.15.28:0/563175104 shutdown_connections 2024-09-21T05:55:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.494+0000 7fc145d9c640 1 -- 172.21.15.28:0/563175104 wait complete. 2024-09-21T05:55:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 Processor -- start 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 -- start start 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1401094f0 0x7fc14019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 0x7fc1401a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc140113b00 con 0x7fc140103bb0 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc140113980 con 0x7fc1401094f0 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc145d9c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc140113c80 con 0x7fc140109e40 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.495+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46188/0 (socket says 172.21.15.28:46188) 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2334899189 learned_addr learned my addr 172.21.15.28:0/2334899189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13effd640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1401094f0 0x7fc14019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 0x7fc1401a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2334899189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 msgr2=0x7fc1401a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 0x7fc1401a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2334899189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1401094f0 msgr2=0x7fc14019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1401094f0 0x7fc14019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2334899189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1401a2520 con 0x7fc140103bb0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 0x7fc1401a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc13000cce0 tx=0x7fc13000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13cff9640 1 -- 172.21.15.28:0/2334899189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13000f650 con 0x7fc140103bb0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1401a2810 con 0x7fc140103bb0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13cff9640 1 -- 172.21.15.28:0/2334899189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13000fca0 con 0x7fc140103bb0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc13cff9640 1 -- 172.21.15.28:0/2334899189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130018b40 con 0x7fc140103bb0 2024-09-21T05:55:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.496+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401a9030 con 0x7fc140103bb0 2024-09-21T05:55:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 Processor -- start 2024-09-21T05:55:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 -- start start 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc13cff9640 1 -- 172.21.15.28:0/2334899189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fc130018ce0 con 0x7fc140103bb0 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b59be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b5a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b08acfea0 con 0x562b087b2070 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562b08abf2d0 con 0x562b08791390 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc14c310740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562b08ac30e0 con 0x562b087891b0 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.499+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46200/0 (socket says 172.21.15.28:46200) 2024-09-21T05:55:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 -- 172.21.15.28:0/261774440 learned_addr learned my addr 172.21.15.28:0/261774440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13effd640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b5a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b59be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 -- 172.21.15.28:0/261774440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 msgr2=0x562b08b59be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b59be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 -- 172.21.15.28:0/261774440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 msgr2=0x562b08b5a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b5a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 -- 172.21.15.28:0/261774440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0881d650 con 0x562b087b2070 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b59be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13ffff640 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc138009900 tx=0x7fc138032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36392c01ee0c797f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13cff9640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 0x7fc12c0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc135ffb640 1 -- 172.21.15.28:0/261774440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13803f070 con 0x562b087b2070 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 0x7fc12c0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc135ffb640 1 -- 172.21.15.28:0/261774440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138031760 con 0x562b087b2070 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.500+0000 7fc135ffb640 1 -- 172.21.15.28:0/261774440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138035070 con 0x562b087b2070 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc13cff9640 1 -- 172.21.15.28:0/2334899189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc1300e7c20 con 0x7fc140103bb0 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 -- 172.21.15.28:0/261774440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 msgr2=0x562b08b596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 0x7fc12c0b8bc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x562b08b58320 tx=0x7fc130000f40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc138009900 tx=0x7fc138032e70 comp rx=0 tx=0).stop 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 -- 172.21.15.28:0/261774440 shutdown_connections 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b5a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b59be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 --2- 172.21.15.28:0/261774440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b596a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.501+0000 7fc14c310740 1 -- 172.21.15.28:0/261774440 >> 172.21.15.28:0/261774440 conn(0x562b087a6460 msgr2=0x562b08b57590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 -- 172.21.15.28:0/261774440 shutdown_connections 2024-09-21T05:55:50.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 -- 172.21.15.28:0/261774440 wait complete. 2024-09-21T05:55:50.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 -- start start 2024-09-21T05:55:50.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 Processor -- start 2024-09-21T05:55:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b64dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b65300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.502+0000 7fc14c310740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc14c310740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b08920700 con 0x562b087b2070 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc14c310740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562b08a5a160 con 0x562b08791390 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc14c310740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562b08b67e20 con 0x562b087891b0 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc14c310740 10 client.0 _ll_register_callbacks cb 0x562b08a739e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46202/0 (socket says 172.21.15.28:46202) 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 -- 172.21.15.28:0/2696713780 learned_addr learned my addr 172.21.15.28:0/2696713780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13effd640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b65300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b64dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 -- 172.21.15.28:0/2696713780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 msgr2=0x562b08b64dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b64dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 -- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 msgr2=0x562b08b65300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b65300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 -- 172.21.15.28:0/2696713780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b08a87c90 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13effd640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b65300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc138005e40 tx=0x7fc138005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94820d777a7e6eeb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc11effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.503+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138035070 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc138004260 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc138031760 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc138057050 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 10 client.17532 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562b08b73ad0 con 0x562b087b2070 2024-09-21T05:55:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138031760 con 0x562b087b2070 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fc13804f020 con 0x562b087b2070 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 10 client.17532 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 10 client.17532 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 20 client.17532 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b087709f0 0x562b08b76e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 4 client.17532 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.504+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b08ad1df0 con 0x562b087709f0 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.505+0000 7fc13effd640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b087709f0 0x562b08b76e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.505+0000 7fc13effd640 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b087709f0 0x562b08b76e30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e77726d2e15df87 server_cookie=e6e369769cb7deb9 in_seq=0 out_seq=0 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.505+0000 7fc11effd640 10 client.17532 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc11effd640 1 -- 172.21.15.28:0/2696713780 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562b08ad1df0 con 0x562b087709f0 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc11effd640 10 client.17532 handle_command_reply: tid=0 2024-09-21T05:55:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 client.17532 shutdown 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 msgr2=0x562b08b71140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc138005e40 tx=0x7fc138005420 comp rx=0 tx=0).stop 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 shutdown_connections 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b087709f0 0x562b08b76e30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087b2070 0x562b08b71140 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b08791390 0x562b08b65300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 --2- 172.21.15.28:0/2696713780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087891b0 0x562b08b64dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc11effd640 0 client.17532 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 >> 172.21.15.28:0/2696713780 conn(0x562b087a6460 msgr2=0x562b08b5c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 shutdown_connections 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 1 -- 172.21.15.28:0/2696713780 wait complete. 2024-09-21T05:55:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.506+0000 7fc14c310740 20 client.17532 trim_cache size 0 max 16384 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 Processor -- start 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 -- start start 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 0x562b08b710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087b2070 0x562b08b61fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401b18c0 con 0x562b087891b0 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401b34f0 con 0x562b087b2070 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc14c310740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401b1ff0 con 0x562b08791390 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46214/0 (socket says 172.21.15.28:46214) 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 -- 172.21.15.28:0/1154105154 learned_addr learned my addr 172.21.15.28:0/1154105154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13effd640 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 0x562b08b710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 -- 172.21.15.28:0/1154105154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 msgr2=0x562b08b710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 0x562b08b710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 -- 172.21.15.28:0/1154105154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087b2070 msgr2=0x562b08b61fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087b2070 0x562b08b61fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13ffff640 1 -- 172.21.15.28:0/1154105154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b08b5c500 con 0x562b087891b0 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.520+0000 7fc13effd640 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 0x562b08b710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc13802a930 tx=0x7fc138005950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7bacf21143969c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc1357fa640 1 -- 172.21.15.28:0/1154105154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138035040 con 0x562b087891b0 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc1357fa640 1 -- 172.21.15.28:0/1154105154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1380035f0 con 0x562b087891b0 2024-09-21T05:55:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc1357fa640 1 -- 172.21.15.28:0/1154105154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13803f070 con 0x562b087891b0 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc14c310740 1 -- 172.21.15.28:0/1154105154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 msgr2=0x562b08b575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.521+0000 7fc14c310740 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc13802a930 tx=0x7fc138005950 comp rx=0 tx=0).stop 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 -- 172.21.15.28:0/1154105154 shutdown_connections 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087b2070 0x562b08b61fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b08791390 0x562b08b710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 --2- 172.21.15.28:0/1154105154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b087891b0 0x562b08b575e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 -- 172.21.15.28:0/1154105154 >> 172.21.15.28:0/1154105154 conn(0x562b087a6460 msgr2=0x562b08b56ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 -- 172.21.15.28:0/1154105154 shutdown_connections 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 -- 172.21.15.28:0/1154105154 wait complete. 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 -- start start 2024-09-21T05:55:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.522+0000 7fc14c310740 1 Processor -- start 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087891b0 0x562b08b7a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087b2070 0x562b08b77ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401b2df0 con 0x562b08791390 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401b26f0 con 0x562b087891b0 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401af8a0 con 0x562b087b2070 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 10 client.0 _ll_register_callbacks cb 0x7fc138032d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46226/0 (socket says 172.21.15.28:46226) 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 -- 172.21.15.28:0/1263187590 learned_addr learned my addr 172.21.15.28:0/1263187590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087891b0 0x562b08b7a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087b2070 0x562b08b77ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 -- 172.21.15.28:0/1263187590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087b2070 msgr2=0x562b08b77ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087b2070 0x562b08b77ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 -- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087891b0 msgr2=0x562b08b7a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087891b0 0x562b08b7a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 -- 172.21.15.28:0/1263187590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b08b78530 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc13effd640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x562b08b67e80 tx=0x7fc12800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c9f398100787e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc11effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc128011440 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562b08acd8a0 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.523+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fc1280115e0 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc1280117e0 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 10 client.17538 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562b08be13e0 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562b08acd8a0 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fc12802ca60 con 0x562b08791390 2024-09-21T05:55:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 10 client.17538 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 10 client.17538 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 20 client.17538 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b08745da0 0x562b08be3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 4 client.17538 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b08acacd0 con 0x562b08745da0 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.524+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b08745da0 0x562b08be3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc13ffff640 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b08745da0 0x562b08be3b70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=550e80c5021765ca server_cookie=d7bd9a1d62af12d0 in_seq=0 out_seq=0 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc11effd640 10 client.17538 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc11effd640 1 -- 172.21.15.28:0/1263187590 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b08acacd0 con 0x562b08745da0 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc11effd640 10 client.17538 handle_command_reply: tid=0 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 client.17538 shutdown 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 msgr2=0x562b08b779b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x562b08b67e80 tx=0x7fc12800bf80 comp rx=0 tx=0).stop 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 shutdown_connections 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x562b08745da0 0x562b08be3b70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562b087b2070 0x562b08b77ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b08791390 0x562b08b779b0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 --2- 172.21.15.28:0/1263187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562b087891b0 0x562b08b7a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 >> 172.21.15.28:0/1263187590 conn(0x562b087a6460 msgr2=0x562b08b5a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc11effd640 0 client.17538 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.525+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 shutdown_connections 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc14c310740 1 -- 172.21.15.28:0/1263187590 wait complete. 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc14c310740 20 client.17538 trim_cache size 0 max 16384 2024-09-21T05:55:50.532 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 msgr2=0x7fc12c0b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 0x7fc12c0b8bc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x562b08b58320 tx=0x7fc130000f40 comp rx=0 tx=0).stop 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 msgr2=0x7fc14019c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc13000cce0 tx=0x7fc13000a620 comp rx=0 tx=0).stop 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 shutdown_connections 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc140109e40 0x7fc1401a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1401094f0 0x7fc14019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fc12c0b66b0 0x7fc12c0b8bc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 --2- 172.21.15.28:0/2334899189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140103bb0 0x7fc14019c670 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:50.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 >> 172.21.15.28:0/2334899189 conn(0x7fc1400ff600 msgr2=0x7fc14010e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:50.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 shutdown_connections 2024-09-21T05:55:50.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:50.526+0000 7fc145d9c640 1 -- 172.21.15.28:0/2334899189 wait complete. 2024-09-21T05:55:50.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:50 smithi129 ceph-mon[34885]: pgmap v593: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:50.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:50 smithi181 ceph-mon[34819]: pgmap v593: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:55:51.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:55:51.646 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 -- 172.21.15.28:0/1403203609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca010de60 msgr2=0x7f1ca0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 --2- 172.21.15.28:0/1403203609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca010de60 0x7f1ca0110330 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009ca0 tx=0x7f1c9802a8e0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 -- 172.21.15.28:0/1403203609 shutdown_connections 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 --2- 172.21.15.28:0/1403203609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca010de60 0x7f1ca0110330 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 --2- 172.21.15.28:0/1403203609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 0x7f1ca010d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 --2- 172.21.15.28:0/1403203609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca0104a30 0x7f1ca0104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 -- 172.21.15.28:0/1403203609 >> 172.21.15.28:0/1403203609 conn(0x7f1ca0079760 msgr2=0x7f1ca0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 -- 172.21.15.28:0/1403203609 shutdown_connections 2024-09-21T05:55:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.110+0000 7f1ca8156640 1 -- 172.21.15.28:0/1403203609 wait complete. 2024-09-21T05:55:52.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 Processor -- start 2024-09-21T05:55:52.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 -- start start 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 0x7f1ca019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca010de60 0x7f1ca019fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca0113b70 con 0x7f1ca0104a30 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01139f0 con 0x7f1ca0105380 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca8156640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca0113cf0 con 0x7f1ca010de60 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46242/0 (socket says 172.21.15.28:46242) 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.111+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/3901682425 learned_addr learned my addr 172.21.15.28:0/3901682425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca010de60 0x7f1ca019fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 0x7f1ca019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/3901682425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca010de60 msgr2=0x7f1ca019fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca010de60 0x7f1ca019fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/3901682425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 msgr2=0x7f1ca019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 0x7f1ca019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/3901682425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca01a0450 con 0x7f1ca0104a30 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d950 tx=0x7f1c9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1c96ffd640 1 -- 172.21.15.28:0/3901682425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c017070 con 0x7f1ca0104a30 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca01a0740 con 0x7f1ca0104a30 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1c96ffd640 1 -- 172.21.15.28:0/3901682425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9c009c30 con 0x7f1ca0104a30 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1c96ffd640 1 -- 172.21.15.28:0/3901682425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00f620 con 0x7f1ca0104a30 2024-09-21T05:55:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.112+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca01a6ea0 con 0x7f1ca0104a30 2024-09-21T05:55:52.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.115+0000 7f1cae6ca740 1 Processor -- start 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.115+0000 7f1cae6ca740 1 -- start start 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcde2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcded40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d3efc54ea0 con 0x55d3ef90e1b0 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d3efc442d0 con 0x55d3ef937070 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d3efc480e0 con 0x55d3ef916390 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1c96ffd640 1 -- 172.21.15.28:0/3901682425 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1c9c01a070 con 0x7f1ca0104a30 2024-09-21T05:55:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcded40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46244/0 (socket says 172.21.15.28:46244) 2024-09-21T05:55:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/2269005317 learned_addr learned my addr 172.21.15.28:0/2269005317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcde2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 -- 172.21.15.28:0/2269005317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 msgr2=0x55d3efcde2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcde2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 -- 172.21.15.28:0/2269005317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 msgr2=0x55d3efcde800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca56f1640 1 -- 172.21.15.28:0/2269005317 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3ef9a2650 con 0x55d3ef916390 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.116+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcded40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006b680 tx=0x7f1c9000bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d9137371b3c1eff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1c96ffd640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 0x7f1c8c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1c8bfff640 1 -- 172.21.15.28:0/2269005317 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000be10 con 0x55d3ef916390 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1c8bfff640 1 -- 172.21.15.28:0/2269005317 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90019070 con 0x55d3ef916390 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1c8bfff640 1 -- 172.21.15.28:0/2269005317 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90014c90 con 0x55d3ef916390 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 0x7f1c8c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.117+0000 7f1c96ffd640 1 -- 172.21.15.28:0/3901682425 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1c9c0e71c0 con 0x7f1ca0104a30 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 -- 172.21.15.28:0/2269005317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 msgr2=0x55d3efcded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 0x7f1c8c0b8b90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009a60 tx=0x7f1c98032fe0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcded40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006b680 tx=0x7f1c9000bbc0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 -- 172.21.15.28:0/2269005317 shutdown_connections 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcded40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcde800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/2269005317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcde2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 -- 172.21.15.28:0/2269005317 >> 172.21.15.28:0/2269005317 conn(0x55d3ef92b460 msgr2=0x55d3efcdc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 -- 172.21.15.28:0/2269005317 shutdown_connections 2024-09-21T05:55:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.118+0000 7f1cae6ca740 1 -- 172.21.15.28:0/2269005317 wait complete. 2024-09-21T05:55:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 -- start start 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 Processor -- start 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d3efaa5700 con 0x55d3ef90e1b0 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d3efbdf160 con 0x55d3ef937070 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d3efcf0e00 con 0x55d3ef916390 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1cae6ca740 10 client.0 _ll_register_callbacks cb 0x55d3efbf89e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46250/0 (socket says 172.21.15.28:46250) 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.119+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/76351628 learned_addr learned my addr 172.21.15.28:0/76351628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 msgr2=0x55d3efcee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcee2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/76351628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 msgr2=0x55d3efcee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/76351628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3efc0cc90 con 0x55d3ef90e1b0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0acfa0 tx=0x7f1c9c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e6a340a6388b94 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1c897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c021030 con 0x55d3ef90e1b0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c90019070 con 0x55d3ef90e1b0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1c9c0a8960 con 0x55d3ef90e1b0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c0a8eb0 con 0x55d3ef90e1b0 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1cae6ca740 10 client.17550 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.120+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55d3efcfcb50 con 0x55d3ef90e1b0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90019070 con 0x55d3ef90e1b0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f1c9c0a8b60 con 0x55d3ef90e1b0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 10 client.17550 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 10 client.17550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 20 client.17550 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef8f59f0 0x55d3efd000a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 4 client.17550 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3efc56df0 con 0x55d3ef8f59f0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.121+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef8f59f0 0x55d3efd000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef8f59f0 0x55d3efd000a0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=591a870cd4c812c1 server_cookie=517b95b64b57046f in_seq=0 out_seq=0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1c897fa640 10 client.17550 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1c897fa640 1 -- 172.21.15.28:0/76351628 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d3efc56df0 con 0x55d3ef8f59f0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1c897fa640 10 client.17550 handle_command_reply: tid=0 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1cae6ca740 1 client.17550 shutdown 2024-09-21T05:55:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.122+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 msgr2=0x55d3efcedda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0acfa0 tx=0x7f1c9c00bed0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 shutdown_connections 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef8f59f0 0x55d3efd000a0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef937070 0x55d3efcee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef916390 0x55d3efcee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1c897fa640 0 client.17550 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/76351628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef90e1b0 0x55d3efcedda0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 >> 172.21.15.28:0/76351628 conn(0x55d3ef92b460 msgr2=0x55d3efcda750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 shutdown_connections 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 1 -- 172.21.15.28:0/76351628 wait complete. 2024-09-21T05:55:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.123+0000 7f1cae6ca740 20 client.17550 trim_cache size 0 max 16384 2024-09-21T05:55:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 Processor -- start 2024-09-21T05:55:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 -- start start 2024-09-21T05:55:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef8f59f0 0x55d3efcee260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef90e1b0 0x55d3efcfa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01ad610 con 0x55d3ef916390 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01af240 con 0x55d3ef90e1b0 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01add40 con 0x55d3ef8f59f0 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.136+0000 7f1ca66f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46260/0 (socket says 172.21.15.28:46260) 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 -- 172.21.15.28:0/1973541989 learned_addr learned my addr 172.21.15.28:0/1973541989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef90e1b0 0x55d3efcfa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef8f59f0 0x55d3efcee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 -- 172.21.15.28:0/1973541989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef8f59f0 msgr2=0x55d3efcee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef8f59f0 0x55d3efcee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 -- 172.21.15.28:0/1973541989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef90e1b0 msgr2=0x55d3efcfa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef90e1b0 0x55d3efcfa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 -- 172.21.15.28:0/1973541989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3efcfa060 con 0x55d3ef916390 2024-09-21T05:55:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009ca0 tx=0x7f1c98041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2e4794e803daa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1c8b7fe640 1 -- 172.21.15.28:0/1973541989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98031ca0 con 0x55d3ef916390 2024-09-21T05:55:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1c8b7fe640 1 -- 172.21.15.28:0/1973541989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c98036040 con 0x55d3ef916390 2024-09-21T05:55:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.137+0000 7f1c8b7fe640 1 -- 172.21.15.28:0/1973541989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98030cb0 con 0x55d3ef916390 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 -- 172.21.15.28:0/1973541989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 msgr2=0x55d3efcfa7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009ca0 tx=0x7f1c98041000 comp rx=0 tx=0).stop 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 -- 172.21.15.28:0/1973541989 shutdown_connections 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efcfa7a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef90e1b0 0x55d3efcfa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/1973541989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef8f59f0 0x55d3efcee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 -- 172.21.15.28:0/1973541989 >> 172.21.15.28:0/1973541989 conn(0x55d3ef92b460 msgr2=0x55d3efcedb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 -- 172.21.15.28:0/1973541989 shutdown_connections 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.138+0000 7f1cae6ca740 1 -- 172.21.15.28:0/1973541989 wait complete. 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 -- start start 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 Processor -- start 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef90e1b0 0x55d3efcf2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efd69af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01aeb40 con 0x55d3ef916390 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01ae440 con 0x55d3ef8f59f0 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca01ab5f0 con 0x55d3ef90e1b0 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1ca5ef2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37268/0 (socket says 172.21.15.28:37268) 2024-09-21T05:55:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/326660575 learned_addr learned my addr 172.21.15.28:0/326660575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1cae6ca740 10 client.0 _ll_register_callbacks cb 0x7f1c9c0a94b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.139+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef90e1b0 0x55d3efcf2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca66f3640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efd69af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef90e1b0 msgr2=0x55d3efcf2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef90e1b0 0x55d3efcf2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/326660575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 msgr2=0x55d3efd69af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efd69af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 -- 172.21.15.28:0/326660575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3efd6a030 con 0x55d3ef8f59f0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1ca5ef2640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0b6bf0 tx=0x7f1c9c010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65bc7010d5d286de server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1c897fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c014020 con 0x55d3ef8f59f0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c98036040 con 0x55d3ef8f59f0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.140+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1c9c0b2c80 con 0x55d3ef8f59f0 2024-09-21T05:55:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c021040 con 0x55d3ef8f59f0 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 10 client.27062 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55d3efd6b260 con 0x55d3ef8f59f0 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c98036040 con 0x55d3ef8f59f0 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f1c9c0b2e80 con 0x55d3ef8f59f0 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 10 client.27062 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 10 client.27062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 20 client.27062 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef937070 0x55d3efd6da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 4 client.27062 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3efc528a0 con 0x55d3ef937070 2024-09-21T05:55:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.141+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef937070 0x55d3efd6da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1ca56f1640 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef937070 0x55d3efd6da80 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466fb63f8691d13 server_cookie=cecefc6132fdfba9 in_seq=0 out_seq=0 2024-09-21T05:55:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1c897fa640 10 client.27062 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1c897fa640 1 -- 172.21.15.28:0/326660575 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3efc528a0 con 0x55d3ef937070 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1c897fa640 10 client.27062 handle_command_reply: tid=0 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 client.27062 shutdown 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 msgr2=0x55d3efcf2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0b6bf0 tx=0x7f1c9c010c40 comp rx=0 tx=0).stop 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 shutdown_connections 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55d3ef937070 0x55d3efd6da80 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3ef916390 0x55d3efd69af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3ef90e1b0 0x55d3efcf2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 --2- 172.21.15.28:0/326660575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d3ef8f59f0 0x55d3efcf2790 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 >> 172.21.15.28:0/326660575 conn(0x55d3ef92b460 msgr2=0x55d3efcfa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1c897fa640 0 client.27062 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 shutdown_connections 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 1 -- 172.21.15.28:0/326660575 wait complete. 2024-09-21T05:55:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.142+0000 7f1cae6ca740 20 client.27062 trim_cache size 0 max 16384 2024-09-21T05:55:52.150 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 msgr2=0x7f1c8c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 0x7f1c8c0b8b90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009a60 tx=0x7f1c98032fe0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 msgr2=0x7f1ca019a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d950 tx=0x7f1c9c00de10 comp rx=0 tx=0).stop 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 shutdown_connections 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ca010de60 0x7f1ca019fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ca0105380 0x7f1ca019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1c8c0b6680 0x7f1c8c0b8b90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 --2- 172.21.15.28:0/3901682425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ca0104a30 0x7f1ca019a4e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 >> 172.21.15.28:0/3901682425 conn(0x7f1ca0079760 msgr2=0x7f1ca00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 shutdown_connections 2024-09-21T05:55:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:52.143+0000 7f1ca8156640 1 -- 172.21.15.28:0/3901682425 wait complete. 2024-09-21T05:55:52.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:52 smithi028 ceph-mon[31227]: pgmap v594: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:52.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:52 smithi129 ceph-mon[34885]: pgmap v594: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:52.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:52 smithi181 ceph-mon[34819]: pgmap v594: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:55:54.051 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:55:54.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:54.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:54 smithi028 ceph-mon[31227]: pgmap v595: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:54.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:55:54.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:55:54.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:55:54.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:55:54.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:54 smithi181 ceph-mon[34819]: pgmap v595: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:54.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:55:54.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:55:54.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:55:54.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:55:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:54 smithi129 ceph-mon[34885]: pgmap v595: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:55:54.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:55:54.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:55:54.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:55:54.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:55:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 -- 172.21.15.28:0/3236145170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 msgr2=0x7f3b041106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/3236145170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b041106b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009900 tx=0x7f3afc02a410 comp rx=0 tx=0).stop 2024-09-21T05:55:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 -- 172.21.15.28:0/3236145170 shutdown_connections 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/3236145170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b041106b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/3236145170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b041062d0 0x7f3b041066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/3236145170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b04102080 0x7f3b04102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 -- 172.21.15.28:0/3236145170 >> 172.21.15.28:0/3236145170 conn(0x7f3b04079f90 msgr2=0x7f3b0407a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 -- 172.21.15.28:0/3236145170 shutdown_connections 2024-09-21T05:55:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.807+0000 7f3b0c67d640 1 -- 172.21.15.28:0/3236145170 wait complete. 2024-09-21T05:55:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 Processor -- start 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 -- start start 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b041062d0 0x7f3b0419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b0419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113b10 con 0x7f3b04106c20 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113990 con 0x7f3b04102080 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0c67d640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b04113c90 con 0x7f3b041062d0 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0a419640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0a419640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:37284/0 (socket says 172.21.15.28:37284) 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0a419640 1 -- 172.21.15.28:0/1247547589 learned_addr learned my addr 172.21.15.28:0/1247547589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b0419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.808+0000 7f3b09c18640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b041062d0 0x7f3b0419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 -- 172.21.15.28:0/1247547589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b041062d0 msgr2=0x7f3b0419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b041062d0 0x7f3b0419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 -- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 msgr2=0x7f3b0419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b0419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 -- 172.21.15.28:0/1247547589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b041a0450 con 0x7f3b04102080 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0a419640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3af400eaa0 tx=0x7f3af400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b037fe640 1 -- 172.21.15.28:0/1247547589 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4016070 con 0x7f3b04102080 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b041a0740 con 0x7f3b04102080 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b037fe640 1 -- 172.21.15.28:0/1247547589 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3af4003e20 con 0x7f3b04102080 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b037fe640 1 -- 172.21.15.28:0/1247547589 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af40116d0 con 0x7f3b04102080 2024-09-21T05:55:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.809+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b041a6ea0 con 0x7f3b04102080 2024-09-21T05:55:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 Processor -- start 2024-09-21T05:55:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 -- start start 2024-09-21T05:55:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e303363430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303363970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e303363eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3032d9e60 con 0x55e302f9b670 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e3032c9290 con 0x55e302f93490 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e3032cd0a0 con 0x55e302fbc350 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.812+0000 7f3b037fe640 1 -- 172.21.15.28:0/1247547589 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3af4019070 con 0x7f3b04102080 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b09c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e303363eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e303363430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b09c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e303363eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46310/0 (socket says 172.21.15.28:46310) 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b09c18640 1 -- 172.21.15.28:0/698890127 learned_addr learned my addr 172.21.15.28:0/698890127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0a419640 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303363970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/698890127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 msgr2=0x55e303363970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303363970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/698890127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 msgr2=0x55e303363eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e303363eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/698890127 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e303027930 con 0x55e302fbc350 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0a419640 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303363970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e303363430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3afc004770 tx=0x7f3afc031b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6badacf2ffb60df0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.813+0000 7f3b00ff9640 1 -- 172.21.15.28:0/698890127 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc040070 con 0x55e302fbc350 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b00ff9640 1 -- 172.21.15.28:0/698890127 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc004030 con 0x55e302fbc350 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b00ff9640 1 -- 172.21.15.28:0/698890127 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc035040 con 0x55e302fbc350 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b037fe640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 0x7f3aec0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b0a419640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 0x7f3aec0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b037fe640 1 -- 172.21.15.28:0/1247547589 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3af40e77b0 con 0x7f3b04102080 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 -- 172.21.15.28:0/698890127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 msgr2=0x55e303363430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b0a419640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 0x7f3aec0b8b90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55e3033620b0 tx=0x7f3af40044f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e303363430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3afc004770 tx=0x7f3afc031b50 comp rx=0 tx=0).stop 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 -- 172.21.15.28:0/698890127 shutdown_connections 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e303363eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303363970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/698890127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e303363430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.814+0000 7f3b12bf1740 1 -- 172.21.15.28:0/698890127 >> 172.21.15.28:0/698890127 conn(0x55e302fb0740 msgr2=0x55e303361320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- 172.21.15.28:0/698890127 shutdown_connections 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- 172.21.15.28:0/698890127 wait complete. 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- start start 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 Processor -- start 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 0x55e30336f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302fbc350 0x55e30337aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e30312a890 con 0x55e302f93490 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e303263d40 con 0x55e302fbc350 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.815+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e303371bb0 con 0x55e302f9b670 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b12bf1740 10 client.0 _ll_register_callbacks cb 0x55e3032c7c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46324/0 (socket says 172.21.15.28:46324) 2024-09-21T05:55:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/2173872324 learned_addr learned my addr 172.21.15.28:0/2173872324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b09c18640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 0x55e30336f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302fbc350 0x55e30337aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 msgr2=0x55e30336f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 0x55e30336f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/2173872324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302fbc350 msgr2=0x55e30337aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302fbc350 0x55e30337aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/2173872324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e30337b460 con 0x55e302f93490 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b09c18640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 0x55e30336f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0098d0 tx=0x7f3afc02ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844cf0cd9d791727 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc040070 con 0x55e302f93490 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3afc031430 con 0x55e302f93490 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.816+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3afc004030 con 0x55e302f93490 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3afc058050 con 0x55e302f93490 2024-09-21T05:55:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 10 client.17568 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e30337dd90 con 0x55e302f93490 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc004030 con 0x55e302f93490 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f3afc0035f0 con 0x55e302f93490 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 10 client.17568 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 20 client.17568 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f7acd0 0x55e3033810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 4 client.17568 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3032dbdb0 con 0x55e302f7acd0 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.817+0000 7f3b09c18640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f7acd0 0x55e3033810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.818+0000 7f3b09c18640 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f7acd0 0x55e3033810f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34bedbcd69a66b4e server_cookie=68367af200ab1318 in_seq=0 out_seq=0 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.818+0000 7f3ae9ffb640 10 client.17568 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/2173872324 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e3032dbdb0 con 0x55e302f7acd0 2024-09-21T05:55:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3ae9ffb640 10 client.17568 handle_command_reply: tid=0 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 client.17568 shutdown 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 msgr2=0x55e30336efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0098d0 tx=0x7f3afc02ab90 comp rx=0 tx=0).stop 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 shutdown_connections 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f7acd0 0x55e3033810f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302fbc350 0x55e30337aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3ae9ffb640 0 client.17568 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f9b670 0x55e30336f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/2173872324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f93490 0x55e30336efc0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 >> 172.21.15.28:0/2173872324 conn(0x55e302fb0740 msgr2=0x55e3033665c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 shutdown_connections 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 1 -- 172.21.15.28:0/2173872324 wait complete. 2024-09-21T05:55:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.819+0000 7f3b12bf1740 20 client.17568 trim_cache size 0 max 16384 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 Processor -- start 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 -- start start 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f93490 0x55e303369550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f9b670 0x55e303364a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041af6d0 con 0x55e302fbc350 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041b1300 con 0x55e302f9b670 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041afe00 con 0x55e302f93490 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b0ac1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46334/0 (socket says 172.21.15.28:46334) 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/3574233690 learned_addr learned my addr 172.21.15.28:0/3574233690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b09c18640 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f9b670 0x55e303364a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.837+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f93490 0x55e303369550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/3574233690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f93490 msgr2=0x55e303369550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f93490 0x55e303369550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/3574233690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f9b670 msgr2=0x55e303364a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f9b670 0x55e303364a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 -- 172.21.15.28:0/3574233690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e30336dd20 con 0x55e302fbc350 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0096c0 tx=0x7f3afc009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0c07af616ad1cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3aebfff640 1 -- 172.21.15.28:0/3574233690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc040070 con 0x55e302fbc350 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3aebfff640 1 -- 172.21.15.28:0/3574233690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc04e040 con 0x55e302fbc350 2024-09-21T05:55:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.838+0000 7f3aebfff640 1 -- 172.21.15.28:0/3574233690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc035070 con 0x55e302fbc350 2024-09-21T05:55:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- 172.21.15.28:0/3574233690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 msgr2=0x55e303364f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0096c0 tx=0x7f3afc009650 comp rx=0 tx=0).stop 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- 172.21.15.28:0/3574233690 shutdown_connections 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302fbc350 0x55e303364f60 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f9b670 0x55e303364a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/3574233690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302f93490 0x55e303369550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- 172.21.15.28:0/3574233690 >> 172.21.15.28:0/3574233690 conn(0x55e302fb0740 msgr2=0x55e303364310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- 172.21.15.28:0/3574233690 shutdown_connections 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- 172.21.15.28:0/3574233690 wait complete. 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 -- start start 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.839+0000 7f3b12bf1740 1 Processor -- start 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303382b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e3033e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041b0c00 con 0x55e302f9b670 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041b0500 con 0x55e302f93490 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b041ad6b0 con 0x55e302fbc350 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46342/0 (socket says 172.21.15.28:46342) 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 -- 172.21.15.28:0/1382024397 learned_addr learned my addr 172.21.15.28:0/1382024397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b12bf1740 10 client.0 _ll_register_callbacks cb 0x7f3af80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303382b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e3033e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 -- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 msgr2=0x55e3033e48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e3033e48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 -- 172.21.15.28:0/1382024397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 msgr2=0x55e303382b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303382b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b09c18640 1 -- 172.21.15.28:0/1382024397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3033e4f20 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.840+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303382b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b09c18640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x55e303371bb0 tx=0x7f3af8012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad2582743ee5ac server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af802c060 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3af801ea90 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3afc04e040 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3af801ec90 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 10 client.17574 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e3033ec3e0 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3afc04e040 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f3af8010430 con 0x55e302f9b670 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 10 client.17574 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 20 client.17574 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f50080 0x55e3033eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 4 client.17574 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.841+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e3032d7860 con 0x55e302f50080 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.842+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f50080 0x55e3033eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.842+0000 7f3b0ac1a640 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f50080 0x55e3033eebb0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e25de332fa6a8013 server_cookie=54506613837ad938 in_seq=0 out_seq=0 2024-09-21T05:55:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.842+0000 7f3ae9ffb640 10 client.17574 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3ae9ffb640 1 -- 172.21.15.28:0/1382024397 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e3032d7860 con 0x55e302f50080 2024-09-21T05:55:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3ae9ffb640 10 client.17574 handle_command_reply: tid=0 2024-09-21T05:55:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 client.17574 shutdown 2024-09-21T05:55:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 msgr2=0x55e3033830f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x55e303371bb0 tx=0x7f3af8012eb0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 shutdown_connections 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55e302f50080 0x55e3033eebb0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e302fbc350 0x55e3033e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e302f9b670 0x55e3033830f0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 --2- 172.21.15.28:0/1382024397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e302f93490 0x55e303382b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 >> 172.21.15.28:0/1382024397 conn(0x55e302fb0740 msgr2=0x55e303364cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3ae9ffb640 0 client.17574 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 shutdown_connections 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 1 -- 172.21.15.28:0/1382024397 wait complete. 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.844+0000 7f3b12bf1740 20 client.17574 trim_cache size 0 max 16384 2024-09-21T05:55:54.851 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "dcf8e535-cb94-40fa-83e5-e2ec6e2e97ed", 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 msgr2=0x7f3aec0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 0x7f3aec0b8b90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55e3033620b0 tx=0x7f3af40044f0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 msgr2=0x7f3b0419a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3af400eaa0 tx=0x7f3af400ef60 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 shutdown_connections 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3aec0b6680 0x7f3aec0b8b90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04106c20 0x7f3b0419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b041062d0 0x7f3b0419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 --2- 172.21.15.28:0/1247547589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b04102080 0x7f3b0419a4e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 >> 172.21.15.28:0/1247547589 conn(0x7f3b04079f90 msgr2=0x7f3b04107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 shutdown_connections 2024-09-21T05:55:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:54.845+0000 7f3b0c67d640 1 -- 172.21.15.28:0/1247547589 wait complete. 2024-09-21T05:55:55.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:55 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:55:55.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:55 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:55:55.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:55 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:55 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:55:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:55 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:55:55.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:55 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:55:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:55 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:55:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:55 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:55:55.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:55 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:55:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 -- 172.21.15.28:0/891007546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c10de10 msgr2=0x7f066c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 --2- 172.21.15.28:0/891007546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c10de10 0x7f066c110300 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0664009ca0 tx=0x7f066402a8c0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 -- 172.21.15.28:0/891007546 shutdown_connections 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 --2- 172.21.15.28:0/891007546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c10de10 0x7f066c110300 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 --2- 172.21.15.28:0/891007546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c105300 0x7f066c10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 --2- 172.21.15.28:0/891007546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c1049b0 0x7f066c104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 -- 172.21.15.28:0/891007546 >> 172.21.15.28:0/891007546 conn(0x7f066c079730 msgr2=0x7f066c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 -- 172.21.15.28:0/891007546 shutdown_connections 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.412+0000 7f06740e0640 1 -- 172.21.15.28:0/891007546 wait complete. 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 Processor -- start 2024-09-21T05:55:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 -- start start 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 0x7f066c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 0x7f066c1a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c115c80 con 0x7f066c1049b0 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f066c115b00 con 0x7f066c105300 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f06740e0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f066c115e00 con 0x7f066c10de10 2024-09-21T05:55:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f0671e7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f0671e7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47104/0 (socket says 172.21.15.28:47104) 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.413+0000 7f0671e7c640 1 -- 172.21.15.28:0/3354106063 learned_addr learned my addr 172.21.15.28:0/3354106063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f067267d640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 0x7f066c1a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f067167b640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 0x7f066c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 -- 172.21.15.28:0/3354106063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 msgr2=0x7f066c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 0x7f066c1a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 -- 172.21.15.28:0/3354106063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 msgr2=0x7f066c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 0x7f066c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 -- 172.21.15.28:0/3354106063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066c1a2560 con 0x7f066c1049b0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f067167b640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 0x7f066c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f067267d640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 0x7f066c1a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f066800d770 tx=0x7f066800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0662ffd640 1 -- 172.21.15.28:0/3354106063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668017070 con 0x7f066c1049b0 2024-09-21T05:55:56.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0662ffd640 1 -- 172.21.15.28:0/3354106063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0668009c30 con 0x7f066c1049b0 2024-09-21T05:55:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066c1a2850 con 0x7f066c1049b0 2024-09-21T05:55:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f0662ffd640 1 -- 172.21.15.28:0/3354106063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668003cc0 con 0x7f066c1049b0 2024-09-21T05:55:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.414+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066c1a9070 con 0x7f066c1049b0 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 Processor -- start 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 -- start start 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f0662ffd640 1 -- 172.21.15.28:0/3354106063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f0668003e60 con 0x7f066c1049b0 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a548f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a548fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648a5405e50 con 0x5648a50e8070 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5648a53f5280 con 0x5648a50bf1b0 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067a654740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5648a53f9090 con 0x5648a50c7390 2024-09-21T05:55:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47106/0 (socket says 172.21.15.28:47106) 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067267d640 1 -- 172.21.15.28:0/3256381906 learned_addr learned my addr 172.21.15.28:0/3256381906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.417+0000 7f067167b640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a548fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a548f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 -- 172.21.15.28:0/3256381906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 msgr2=0x5648a548fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a548fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 -- 172.21.15.28:0/3256381906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 msgr2=0x5648a548f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a548f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 -- 172.21.15.28:0/3256381906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a5153650 con 0x5648a50e8070 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a548f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f0662ffd640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 0x7f06580b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f067267d640 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0664006010 tx=0x7f0664009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333f44cab07e39e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.418+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 0x7f06580b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f0657fff640 1 -- 172.21.15.28:0/3256381906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664031d70 con 0x5648a50e8070 2024-09-21T05:55:56.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f0662ffd640 1 -- 172.21.15.28:0/3354106063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f06680e7140 con 0x7f066c1049b0 2024-09-21T05:55:56.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f0657fff640 1 -- 172.21.15.28:0/3256381906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0664036070 con 0x5648a50e8070 2024-09-21T05:55:56.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f0657fff640 1 -- 172.21.15.28:0/3256381906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06640042a0 con 0x5648a50e8070 2024-09-21T05:55:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f0671e7c640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 0x7f06580b8bc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5648a548def0 tx=0x7f066800de80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.419+0000 7f067a654740 1 -- 172.21.15.28:0/3256381906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 msgr2=0x5648a548f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0664006010 tx=0x7f0664009880 comp rx=0 tx=0).stop 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 -- 172.21.15.28:0/3256381906 shutdown_connections 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a548fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a548f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 --2- 172.21.15.28:0/3256381906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a548f270 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 -- 172.21.15.28:0/3256381906 >> 172.21.15.28:0/3256381906 conn(0x5648a50dc460 msgr2=0x5648a548d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 -- 172.21.15.28:0/3256381906 shutdown_connections 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 -- 172.21.15.28:0/3256381906 wait complete. 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 -- start start 2024-09-21T05:55:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.420+0000 7f067a654740 1 Processor -- start 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a549a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50e8070 0x5648a54a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648a5256700 con 0x5648a50c7390 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5648a5390110 con 0x5648a50bf1b0 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5648a549d9f0 con 0x5648a50e8070 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067a654740 10 client.0 _ll_register_callbacks cb 0x5648a53a9990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47108/0 (socket says 172.21.15.28:47108) 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 -- 172.21.15.28:0/2113057666 learned_addr learned my addr 172.21.15.28:0/2113057666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067267d640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a549a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067267d640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50e8070 0x5648a54a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 -- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50e8070 msgr2=0x5648a54a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50e8070 0x5648a54a6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 -- 172.21.15.28:0/2113057666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 msgr2=0x5648a549a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a549a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 -- 172.21.15.28:0/2113057666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a53bdc40 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f067167b640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f065c010820 tx=0x7f065c010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb397bf909110e36 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.421+0000 7f06557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c01d060 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f065c00eda0 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0664036070 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f065c018430 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 10 client.17586 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5648a54a96a0 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0664036070 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f065c0206b0 con 0x5648a50c7390 2024-09-21T05:55:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 10 client.17586 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 20 client.17586 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a50a69f0 0x5648a54acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 4 client.17586 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.422+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648a5407da0 con 0x5648a50a69f0 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.423+0000 7f067267d640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a50a69f0 0x5648a54acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.423+0000 7f067267d640 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a50a69f0 0x5648a54acbd0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54009f9519d8bcd5 server_cookie=45d0c394826ed4a3 in_seq=0 out_seq=0 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.423+0000 7f06557fa640 10 client.17586 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f06557fa640 1 -- 172.21.15.28:0/2113057666 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5648a5407da0 con 0x5648a50a69f0 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f06557fa640 10 client.17586 handle_command_reply: tid=0 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 client.17586 shutdown 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 msgr2=0x5648a549aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f065c010820 tx=0x7f065c010ce0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 shutdown_connections 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a50a69f0 0x5648a54acbd0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50e8070 0x5648a54a6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50c7390 0x5648a549aea0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 --2- 172.21.15.28:0/2113057666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a549a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 >> 172.21.15.28:0/2113057666 conn(0x5648a50dc460 msgr2=0x5648a5492450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f06557fa640 0 client.17586 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 shutdown_connections 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 1 -- 172.21.15.28:0/2113057666 wait complete. 2024-09-21T05:55:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.424+0000 7f067a654740 20 client.17586 trim_cache size 0 max 16384 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 Processor -- start 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 -- start start 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a5492a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a5497b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1b1a30 con 0x5648a50e8070 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1b3660 con 0x5648a50bf1b0 2024-09-21T05:55:56.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067a654740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1b2160 con 0x5648a50c7390 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47122/0 (socket says 172.21.15.28:47122) 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 -- 172.21.15.28:0/4248928873 learned_addr learned my addr 172.21.15.28:0/4248928873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a5492a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067167b640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a5497b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 -- 172.21.15.28:0/4248928873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 msgr2=0x5648a5497b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a5497b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 -- 172.21.15.28:0/4248928873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 msgr2=0x5648a5492a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a5492a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067267d640 1 -- 172.21.15.28:0/4248928873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a548dad0 con 0x5648a50e8070 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.442+0000 7f067167b640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a5497b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f067267d640 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0664003d10 tx=0x7f0664003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48db80ef626f2510 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f06577fe640 1 -- 172.21.15.28:0/4248928873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664032c70 con 0x5648a50e8070 2024-09-21T05:55:56.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f06577fe640 1 -- 172.21.15.28:0/4248928873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0664036070 con 0x5648a50e8070 2024-09-21T05:55:56.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f06577fe640 1 -- 172.21.15.28:0/4248928873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06640046b0 con 0x5648a50e8070 2024-09-21T05:55:56.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f067a654740 1 -- 172.21.15.28:0/4248928873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 msgr2=0x5648a5498090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.443+0000 7f067a654740 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0664003d10 tx=0x7f0664003de0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 -- 172.21.15.28:0/4248928873 shutdown_connections 2024-09-21T05:55:56.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a5498090 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50c7390 0x5648a5497b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 --2- 172.21.15.28:0/4248928873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50bf1b0 0x5648a5492a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 -- 172.21.15.28:0/4248928873 >> 172.21.15.28:0/4248928873 conn(0x5648a50dc460 msgr2=0x5648a5492370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 -- 172.21.15.28:0/4248928873 shutdown_connections 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 -- 172.21.15.28:0/4248928873 wait complete. 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 -- start start 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.444+0000 7f067a654740 1 Processor -- start 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50bf1b0 0x5648a549ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50c7390 0x5648a549f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1b2f60 con 0x5648a50e8070 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1b2860 con 0x5648a50c7390 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1afa10 con 0x5648a50bf1b0 2024-09-21T05:55:56.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067a654740 10 client.0 _ll_register_callbacks cb 0x7f065c00b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47130/0 (socket says 172.21.15.28:47130) 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 -- 172.21.15.28:0/47463350 learned_addr learned my addr 172.21.15.28:0/47463350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50bf1b0 0x5648a549ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067167b640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50c7390 0x5648a549f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 -- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50bf1b0 msgr2=0x5648a549ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50bf1b0 0x5648a549ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 -- 172.21.15.28:0/47463350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50c7390 msgr2=0x5648a549f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50c7390 0x5648a549f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.445+0000 7f067267d640 1 -- 172.21.15.28:0/47463350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a5516750 con 0x5648a50e8070 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067267d640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x5648a549db00 tx=0x7f0664005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a61bfe3a573809f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664031af0 con 0x5648a50e8070 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f0664031c90 con 0x5648a50e8070 2024-09-21T05:55:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0664036070 con 0x5648a50e8070 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0664004430 con 0x5648a50e8070 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 10 client.17592 fetch_fsmap learned FSMap version 48 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5648a55179e0 con 0x5648a50e8070 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0664036070 con 0x5648a50e8070 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f066404f040 con 0x5648a50e8070 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 10 client.17592 fetch_fsmap finished waiting for FSMap version 48 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 10 client.17592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 20 client.17592 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a507bda0 0x5648a5519d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 4 client.17592 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.446+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5648a5403850 con 0x5648a507bda0 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.447+0000 7f067167b640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a507bda0 0x5648a5519d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:56.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.447+0000 7f06557fa640 10 client.17592 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.447+0000 7f067167b640 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a507bda0 0x5648a5519d10 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf0d2d8f985606f server_cookie=d0dbd3315ca0c789 in_seq=0 out_seq=0 2024-09-21T05:55:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f06557fa640 1 -- 172.21.15.28:0/47463350 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3782 (crc 0 0 0) 0x5648a5403850 con 0x5648a507bda0 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f06557fa640 10 client.17592 handle_command_reply: tid=0 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 client.17592 shutdown 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 msgr2=0x5648a549f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x5648a549db00 tx=0x7f0664005d80 comp rx=0 tx=0).stop 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 shutdown_connections 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5648a507bda0 0x5648a5519d10 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a50e8070 0x5648a549f7d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648a50c7390 0x5648a549f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f06557fa640 0 client.17592 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:55:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 --2- 172.21.15.28:0/47463350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648a50bf1b0 0x5648a549ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 >> 172.21.15.28:0/47463350 conn(0x5648a50dc460 msgr2=0x5648a54902a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 shutdown_connections 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.448+0000 7f067a654740 1 -- 172.21.15.28:0/47463350 wait complete. 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f067a654740 20 client.17592 trim_cache size 0 max 16384 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:55:56.456 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:09.151907+0000", 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.457 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.458 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "dcf8e535-cb94-40fa-83e5-e2ec6e2e97ed": { 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:54.843004+0000", 2024-09-21T05:55:56.459 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.460 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:55:56.462 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:55:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 msgr2=0x7f06580b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 0x7f06580b8bc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5648a548def0 tx=0x7f066800de80 comp rx=0 tx=0).stop 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 msgr2=0x7f066c19c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f066800d770 tx=0x7f066800dc30 comp rx=0 tx=0).stop 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 shutdown_connections 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f066c10de10 0x7f066c1a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f066c105300 0x7f066c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f06580b66b0 0x7f06580b8bc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 --2- 172.21.15.28:0/3354106063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c1049b0 0x7f066c19c6b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 >> 172.21.15.28:0/3354106063 conn(0x7f066c079730 msgr2=0x7f066c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 shutdown_connections 2024-09-21T05:55:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:56.449+0000 7f06740e0640 1 -- 172.21.15.28:0/3354106063 wait complete. 2024-09-21T05:55:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:56 smithi129 ceph-mon[34885]: pgmap v596: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:56.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:56 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:55:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:56 smithi181 ceph-mon[34819]: pgmap v596: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:56 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:55:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:56 smithi028 ceph-mon[31227]: pgmap v596: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:55:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:56 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:55:57.448 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:55:57.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:55:57.705 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- 172.21.15.28:0/3743225951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301052d0 msgr2=0x7fa03010d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3743225951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301052d0 0x7fa03010d910 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009900 tx=0x7fa02c02a410 comp rx=0 tx=0).stop 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- 172.21.15.28:0/3743225951 shutdown_connections 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3743225951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 0x7fa0301102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3743225951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301052d0 0x7fa03010d910 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3743225951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301048a0 0x7fa030104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- 172.21.15.28:0/3743225951 >> 172.21.15.28:0/3743225951 conn(0x7fa030079730 msgr2=0x7fa030079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- 172.21.15.28:0/3743225951 shutdown_connections 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- 172.21.15.28:0/3743225951 wait complete. 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 Processor -- start 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.192+0000 7fa037c9f640 1 -- start start 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 0x7fa03019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 0x7fa03019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa030113aa0 con 0x7fa0301048a0 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa030113920 con 0x7fa0301052d0 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa037c9f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa030113c20 con 0x7fa03010de50 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47138/0 (socket says 172.21.15.28:47138) 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 -- 172.21.15.28:0/3574442519 learned_addr learned my addr 172.21.15.28:0/3574442519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa03623c640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 0x7fa03019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa03523a640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 0x7fa03019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 -- 172.21.15.28:0/3574442519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 msgr2=0x7fa03019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 0x7fa03019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 -- 172.21.15.28:0/3574442519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 msgr2=0x7fa03019aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 0x7fa03019aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 -- 172.21.15.28:0/3574442519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0301a0440 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa03523a640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 0x7fa03019aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.193+0000 7fa035a3b640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa02400d770 tx=0x7fa02400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.194+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024017070 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.194+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa024009c30 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.194+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024003cc0 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.194+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0301a0730 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.194+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0301a6e90 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.195+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa030104cb0 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.197+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fa024003e60 con 0x7fa0301048a0 2024-09-21T05:55:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.198+0000 7fa022ffd640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 0x7fa01c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.198+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fa0240e74c0 con 0x7fa0301048a0 2024-09-21T05:55:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.198+0000 7fa03523a640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 0x7fa01c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.198+0000 7fa03523a640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 0x7fa01c0b8b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa02c000c00 tx=0x7fa02c032da0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.200+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa024009de0 con 0x7fa0301048a0 2024-09-21T05:55:58.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.426+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa03019b600 con 0x7fa0301048a0 2024-09-21T05:55:58.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:58 smithi028 ceph-mon[31227]: pgmap v597: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:55:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.427+0000 7fa022ffd640 1 -- 172.21.15.28:0/3574442519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11993 (secure 0 0 0) 0x7fa0240ada10 con 0x7fa0301048a0 2024-09-21T05:55:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:55:58.430 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"btime":"2024-09-21T05:55:20:235633+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:55:19.279395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[0,1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:55:58.431 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 48 2024-09-21T05:55:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 msgr2=0x7fa01c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 0x7fa01c0b8b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa02c000c00 tx=0x7fa02c032da0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 msgr2=0x7fa03019a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa02400d770 tx=0x7fa02400dc30 comp rx=0 tx=0).stop 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 shutdown_connections 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa03010de50 0x7fa03019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0301052d0 0x7fa03019aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fa01c0b6680 0x7fa01c0b8b90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 --2- 172.21.15.28:0/3574442519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0301048a0 0x7fa03019a4d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 >> 172.21.15.28:0/3574442519 conn(0x7fa030079730 msgr2=0x7fa030100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 shutdown_connections 2024-09-21T05:55:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:58.429+0000 7fa037c9f640 1 -- 172.21.15.28:0/3574442519 wait complete. 2024-09-21T05:55:58.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:58 smithi129 ceph-mon[34885]: pgmap v597: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:55:58.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:58 smithi181 ceph-mon[34819]: pgmap v597: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-21T05:55:59.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:55:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:55:59.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:55:59 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3574442519' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:59.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:55:59 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3574442519' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:59.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:55:59 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3574442519' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:55:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2348737921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 msgr2=0x7f1cb4112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2348737921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb4112620 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009ca0 tx=0x7f1cac02a950 comp rx=0 tx=0).stop 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2348737921 shutdown_connections 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2348737921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb4112620 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2348737921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 0x7f1cb410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2348737921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 0x7f1cb4102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2348737921 >> 172.21.15.28:0/2348737921 conn(0x7f1cb4079740 msgr2=0x7f1cb4079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2348737921 shutdown_connections 2024-09-21T05:55:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.931+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2348737921 wait complete. 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.932+0000 7f1cb8e9c640 1 Processor -- start 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.932+0000 7f1cb8e9c640 1 -- start start 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 0x7f1cb419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 0x7f1cb419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4115c70 con 0x7f1cb4110200 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4115af0 con 0x7f1cb4102fa0 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb8e9c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4115df0 con 0x7f1cb4102650 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47156/0 (socket says 172.21.15.28:47156) 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 -- 172.21.15.28:0/2955336491 learned_addr learned my addr 172.21.15.28:0/2955336491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 0x7f1cb419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 0x7f1cb419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 -- 172.21.15.28:0/2955336491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 msgr2=0x7f1cb419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 0x7f1cb419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 -- 172.21.15.28:0/2955336491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 msgr2=0x7f1cb419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 0x7f1cb419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 -- 172.21.15.28:0/2955336491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb41a2550 con 0x7f1cb4110200 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.933+0000 7f1cb2d9d640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1cac02ae70 tx=0x7f1cac02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.934+0000 7f1cab7fe640 1 -- 172.21.15.28:0/2955336491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac034040 con 0x7f1cb4110200 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.934+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb41a27e0 con 0x7f1cb4110200 2024-09-21T05:55:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.934+0000 7f1cab7fe640 1 -- 172.21.15.28:0/2955336491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cac0308f0 con 0x7f1cb4110200 2024-09-21T05:55:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.934+0000 7f1cab7fe640 1 -- 172.21.15.28:0/2955336491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0047f0 con 0x7f1cb4110200 2024-09-21T05:55:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.934+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb41a8fa0 con 0x7f1cb4110200 2024-09-21T05:55:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.936+0000 7f1cbf410740 1 Processor -- start 2024-09-21T05:55:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.936+0000 7f1cab7fe640 1 -- 172.21.15.28:0/2955336491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1cac037020 con 0x7f1cb4110200 2024-09-21T05:55:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 -- start start 2024-09-21T05:55:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1c8256ea0 con 0x55e1c7f101b0 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e1c82462d0 con 0x55e1c7f18390 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cbf410740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e1c824a0e0 con 0x55e1c7f39070 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cb259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47170/0 (socket says 172.21.15.28:47170) 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cb259c640 1 -- 172.21.15.28:0/2832471380 learned_addr learned my addr 172.21.15.28:0/2832471380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.937+0000 7f1cab7fe640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 0x7f1ca00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 -- 172.21.15.28:0/2832471380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 msgr2=0x55e1c82e02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82e02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 -- 172.21.15.28:0/2832471380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 msgr2=0x55e1c82e0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82e0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 -- 172.21.15.28:0/2832471380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c7fa4650 con 0x55e1c7f101b0 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb2d9d640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 0x7f1ca00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cab7fe640 1 -- 172.21.15.28:0/2955336491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1cac1085e0 con 0x7f1cb4110200 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400bd10 tx=0x7f1ca40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff60b667cf4fca27 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1ca8ff9640 1 -- 172.21.15.28:0/2832471380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4004030 con 0x55e1c7f101b0 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1ca8ff9640 1 -- 172.21.15.28:0/2832471380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca40035f0 con 0x55e1c7f101b0 2024-09-21T05:55:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.938+0000 7f1ca8ff9640 1 -- 172.21.15.28:0/2832471380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca40116a0 con 0x55e1c7f101b0 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 -- 172.21.15.28:0/2832471380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 msgr2=0x55e1c82e0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cb2d9d640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 0x7f1ca00b8b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0cb6a0 tx=0x7f1cac0c2c90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400bd10 tx=0x7f1ca40107a0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 -- 172.21.15.28:0/2832471380 shutdown_connections 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e0800 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2832471380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 -- 172.21.15.28:0/2832471380 >> 172.21.15.28:0/2832471380 conn(0x55e1c7f2d460 msgr2=0x55e1c82de1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 -- 172.21.15.28:0/2832471380 shutdown_connections 2024-09-21T05:55:59.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.939+0000 7f1cbf410740 1 -- 172.21.15.28:0/2832471380 wait complete. 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 -- start start 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 Processor -- start 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f101b0 0x55e1c82eb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f18390 0x55e1c82ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1c80a7700 con 0x55e1c7f39070 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e1c81e1160 con 0x55e1c7f101b0 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e1c82eeaa0 con 0x55e1c7f18390 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cbf410740 10 client.0 _ll_register_callbacks cb 0x55e1c81fa9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47176/0 (socket says 172.21.15.28:47176) 2024-09-21T05:55:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.940+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/2441811693 learned_addr learned my addr 172.21.15.28:0/2441811693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f101b0 0x55e1c82eb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f18390 0x55e1c82ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/2441811693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f18390 msgr2=0x55e1c82ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f18390 0x55e1c82ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f101b0 msgr2=0x55e1c82eb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f101b0 0x55e1c82eb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/2441811693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c820ec90 con 0x55e1c7f39070 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c014dd0 tx=0x7f1c9c016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c631946cacbd2b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1c99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c014110 con 0x55e1c7f39070 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1c9c0142b0 con 0x55e1c7f39070 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca40035f0 con 0x55e1c7f39070 2024-09-21T05:55:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c012450 con 0x55e1c7f39070 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cbf410740 10 client.17610 fetch_fsmap learned FSMap version 49 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.941+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e1c82fa7b0 con 0x55e1c7f39070 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca40035f0 con 0x55e1c7f39070 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f1c9c029640 con 0x55e1c7f39070 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 10 client.17610 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 10 client.17610 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 20 client.17610 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7ef79f0 0x55e1c82fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 4 client.17610 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1c8258df0 con 0x55e1c7ef79f0 2024-09-21T05:55:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.942+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7ef79f0 0x55e1c82fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.943+0000 7f1cb259c640 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7ef79f0 0x55e1c82fdcf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2cc9b2c3a8c86840 server_cookie=a6c30e9a04ddbec3 in_seq=0 out_seq=0 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.943+0000 7f1c99ffb640 10 client.17610 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.943+0000 7f1c99ffb640 1 -- 172.21.15.28:0/2441811693 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e1c8258df0 con 0x55e1c7ef79f0 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.943+0000 7f1c99ffb640 10 client.17610 handle_command_reply: tid=0 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 client.17610 shutdown 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 msgr2=0x55e1c82ec3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c014dd0 tx=0x7f1c9c016040 comp rx=0 tx=0).stop 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 shutdown_connections 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7ef79f0 0x55e1c82fdcf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82ec3e0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f18390 0x55e1c82ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1c99ffb640 0 client.17610 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:59.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 --2- 172.21.15.28:0/2441811693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f101b0 0x55e1c82eb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 >> 172.21.15.28:0/2441811693 conn(0x55e1c7f2d460 msgr2=0x55e1c82e3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 shutdown_connections 2024-09-21T05:55:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 1 -- 172.21.15.28:0/2441811693 wait complete. 2024-09-21T05:55:59.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.944+0000 7f1cbf410740 20 client.17610 trim_cache size 0 max 16384 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 Processor -- start 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 -- start start 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82ed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82dd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b18f0 con 0x55e1c7f101b0 2024-09-21T05:55:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b34f0 con 0x55e1c7f18390 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.957+0000 7f1cbf410740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b1ff0 con 0x55e1c7f39070 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47182/0 (socket says 172.21.15.28:47182) 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/3210023034 learned_addr learned my addr 172.21.15.28:0/3210023034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb259c640 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/3210023034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 msgr2=0x55e1c82dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/3210023034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 msgr2=0x55e1c82ed570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/3210023034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c82e9e40 con 0x55e1c7f101b0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c005f50 tx=0x7f1c9c005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34336b716713126 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1c9bfff640 1 -- 172.21.15.28:0/3210023034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c014110 con 0x55e1c7f101b0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1c9bfff640 1 -- 172.21.15.28:0/3210023034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9c0142b0 con 0x55e1c7f101b0 2024-09-21T05:55:59.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.958+0000 7f1c9bfff640 1 -- 172.21.15.28:0/3210023034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0052a0 con 0x55e1c7f101b0 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- 172.21.15.28:0/3210023034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 msgr2=0x55e1c82e1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c005f50 tx=0x7f1c9c005870 comp rx=0 tx=0).stop 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- 172.21.15.28:0/3210023034 shutdown_connections 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f39070 0x55e1c82dd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82ed570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 --2- 172.21.15.28:0/3210023034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f101b0 0x55e1c82e1cb0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- 172.21.15.28:0/3210023034 >> 172.21.15.28:0/3210023034 conn(0x55e1c7f2d460 msgr2=0x55e1c82ece60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- 172.21.15.28:0/3210023034 shutdown_connections 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- 172.21.15.28:0/3210023034 wait complete. 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.959+0000 7f1cbf410740 1 -- start start 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 Processor -- start 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f101b0 0x55e1c82efce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82f0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b2df0 con 0x55e1c7f39070 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b26f0 con 0x55e1c7f18390 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41af8d0 con 0x55e1c7f101b0 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cbf410740 10 client.0 _ll_register_callbacks cb 0x7f1c9c00cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47194/0 (socket says 172.21.15.28:47194) 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/698825897 learned_addr learned my addr 172.21.15.28:0/698825897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb259c640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f101b0 0x55e1c82efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/698825897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f101b0 msgr2=0x55e1c82efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f101b0 0x55e1c82efce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 msgr2=0x55e1c82f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82f0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.960+0000 7f1cb1d9b640 1 -- 172.21.15.28:0/698825897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c83676a0 con 0x55e1c7f39070 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1cb1d9b640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00ccc0 tx=0x7f1c9c014460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84345542ff018b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c014110 con 0x55e1c7f39070 2024-09-21T05:55:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f1c9c029480 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c9c0142b0 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1c9c029680 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1cbf410740 10 client.17616 fetch_fsmap learned FSMap version 49 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e1c8368900 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9c0142b0 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.961+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f1c9c032070 con 0x55e1c7f39070 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 10 client.17616 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 10 client.17616 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 20 client.17616 populate_metadata read hostname 'smithi028' 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7eccda0 0x55e1c836b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 4 client.17616 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1c82548a0 con 0x55e1c7eccda0 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cb259c640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7eccda0 0x55e1c836b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:55:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1cb259c640 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7eccda0 0x55e1c836b0a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f83883992bb1679f server_cookie=9a1d22f0754f94a5 in_seq=0 out_seq=0 2024-09-21T05:55:59.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1c99ffb640 10 client.17616 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1c99ffb640 1 -- 172.21.15.28:0/698825897 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1c82548a0 con 0x55e1c7eccda0 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.962+0000 7f1c99ffb640 10 client.17616 handle_command_reply: tid=0 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 client.17616 shutdown 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 msgr2=0x55e1c82f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00ccc0 tx=0x7f1c9c014460 comp rx=0 tx=0).stop 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 shutdown_connections 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x55e1c7eccda0 0x55e1c836b0a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1c7f39070 0x55e1c82f0760 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e1c7f18390 0x55e1c82f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 --2- 172.21.15.28:0/698825897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e1c7f101b0 0x55e1c82efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 >> 172.21.15.28:0/698825897 conn(0x55e1c7f2d460 msgr2=0x55e1c82e1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1c99ffb640 0 client.17616 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:55:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 shutdown_connections 2024-09-21T05:55:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 1 -- 172.21.15.28:0/698825897 wait complete. 2024-09-21T05:55:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.963+0000 7f1cbf410740 20 client.17616 trim_cache size 0 max 16384 2024-09-21T05:55:59.970 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 msgr2=0x7f1ca00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 0x7f1ca00b8b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0cb6a0 tx=0x7f1cac0c2c90 comp rx=0 tx=0).stop 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 msgr2=0x7f1cb41a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1cac02ae70 tx=0x7f1cac02f5d0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 shutdown_connections 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f1ca00b6680 0x7f1ca00b8b90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb4110200 0x7f1cb41a1e20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cb4102fa0 0x7f1cb419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 --2- 172.21.15.28:0/2955336491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cb4102650 0x7f1cb419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:55:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 >> 172.21.15.28:0/2955336491 conn(0x7f1cb4079740 msgr2=0x7f1cb4109d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:55:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 shutdown_connections 2024-09-21T05:55:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:55:59.964+0000 7f1cb8e9c640 1 -- 172.21.15.28:0/2955336491 wait complete. 2024-09-21T05:56:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:00 smithi129 ceph-mon[34885]: pgmap v598: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-21T05:56:00.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:00 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:00.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:00 smithi028 ceph-mon[31227]: pgmap v598: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-21T05:56:00.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:00 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:00.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:56:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:00 smithi181 ceph-mon[34819]: pgmap v598: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-21T05:56:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:00 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:01.062 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 -- 172.21.15.28:0/870253779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4102ae0 msgr2=0x7f3ff4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 --2- 172.21.15.28:0/870253779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4102ae0 0x7f3ff4102ef0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009900 tx=0x7f3fe402a410 comp rx=0 tx=0).stop 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 -- 172.21.15.28:0/870253779 shutdown_connections 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 --2- 172.21.15.28:0/870253779 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4108d20 0x7f3ff410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 --2- 172.21.15.28:0/870253779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 0x7f3ff41087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 --2- 172.21.15.28:0/870253779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4102ae0 0x7f3ff4102ef0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 -- 172.21.15.28:0/870253779 >> 172.21.15.28:0/870253779 conn(0x7f3ff4079760 msgr2=0x7f3ff4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 -- 172.21.15.28:0/870253779 shutdown_connections 2024-09-21T05:56:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.532+0000 7f3ffb186640 1 -- 172.21.15.28:0/870253779 wait complete. 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 Processor -- start 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 -- start start 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4102ae0 0x7f3ff419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 0x7f3ff419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4112990 con 0x7f3ff4108d20 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4112810 con 0x7f3ff4102ae0 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ffb186640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff4112b10 con 0x7f3ff41083d0 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ff9723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.533+0000 7f3ff9723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47202/0 (socket says 172.21.15.28:47202) 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 -- 172.21.15.28:0/2193642216 learned_addr learned my addr 172.21.15.28:0/2193642216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4102ae0 0x7f3ff419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 0x7f3ff419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 -- 172.21.15.28:0/2193642216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 msgr2=0x7f3ff419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 0x7f3ff419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 -- 172.21.15.28:0/2193642216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4102ae0 msgr2=0x7f3ff419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4102ae0 0x7f3ff419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 -- 172.21.15.28:0/2193642216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff41a65a0 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff9723640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00eaa0 tx=0x7f3fec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff1ffb640 1 -- 172.21.15.28:0/2193642216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec00cdb0 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff1ffb640 1 -- 172.21.15.28:0/2193642216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fec005020 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ff41a6890 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ff1ffb640 1 -- 172.21.15.28:0/2193642216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0116d0 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.534+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ff41a6d80 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.537+0000 7f40016fa740 1 Processor -- start 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.537+0000 7f3ff1ffb640 1 -- 172.21.15.28:0/2193642216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3fec0051d0 con 0x7f3ff4108d20 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.537+0000 7f40016fa740 1 -- start start 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b140b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b103b1b0 0x55a8b140b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1381ea0 con 0x55a8b1043390 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a8b13712d0 con 0x55a8b1064070 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f40016fa740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a8b13750e0 con 0x55a8b103b1b0 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47208/0 (socket says 172.21.15.28:47208) 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 -- 172.21.15.28:0/3594173233 learned_addr learned my addr 172.21.15.28:0/3594173233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b103b1b0 0x55a8b140b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b140b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 -- 172.21.15.28:0/3594173233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b103b1b0 msgr2=0x55a8b140b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b103b1b0 0x55a8b140b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 -- 172.21.15.28:0/3594173233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 msgr2=0x55a8b140b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b140b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 -- 172.21.15.28:0/3594173233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b10cf650 con 0x55a8b1043390 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff1ffb640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 0x7f3fdc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 0x7f3fdc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800bfd0 tx=0x7f3fe800b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f94b9f81c2c15c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.538+0000 7f3ff1ffb640 1 -- 172.21.15.28:0/2193642216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3fec0e77c0 con 0x7f3ff4108d20 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f3fdaffd640 1 -- 172.21.15.28:0/3594173233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8004230 con 0x55a8b1043390 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f3fdaffd640 1 -- 172.21.15.28:0/3594173233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe80043d0 con 0x55a8b1043390 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f3fdaffd640 1 -- 172.21.15.28:0/3594173233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe80107b0 con 0x55a8b1043390 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f40016fa740 1 -- 172.21.15.28:0/3594173233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 msgr2=0x55a8b140bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 0x7f3fdc0b8b90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4000c00 tx=0x7f3fe4031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.539+0000 7f40016fa740 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800bfd0 tx=0x7f3fe800b9f0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 -- 172.21.15.28:0/3594173233 shutdown_connections 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1043390 0x55a8b140bd40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b103b1b0 0x55a8b140b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 --2- 172.21.15.28:0/3594173233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b140b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 -- 172.21.15.28:0/3594173233 >> 172.21.15.28:0/3594173233 conn(0x55a8b1058460 msgr2=0x55a8b14091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 -- 172.21.15.28:0/3594173233 shutdown_connections 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 -- 172.21.15.28:0/3594173233 wait complete. 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 -- start start 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.540+0000 7f40016fa740 1 Processor -- start 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b103b1b0 0x55a8b14169b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b1416ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8b11d2700 con 0x55a8b1064070 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a8b130c160 con 0x55a8b103b1b0 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1419aa0 con 0x55a8b1043390 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 10 client.0 _ll_register_callbacks cb 0x55a8b13259e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47216/0 (socket says 172.21.15.28:47216) 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 -- 172.21.15.28:0/1505142636 learned_addr learned my addr 172.21.15.28:0/1505142636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b103b1b0 0x55a8b14169b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b1416ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 -- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 msgr2=0x55a8b1416ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b1416ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 -- 172.21.15.28:0/1505142636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b103b1b0 msgr2=0x55a8b14169b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b103b1b0 0x55a8b14169b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 -- 172.21.15.28:0/1505142636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b1339c90 con 0x55a8b1064070 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3ff9723640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0aa930 tx=0x7f3fec0057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7279824344c9efae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3fd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec01f030 con 0x55a8b1064070 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3fec0acb30 con 0x55a8b1064070 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.541+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe80043d0 con 0x55a8b1064070 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3fec0b7030 con 0x55a8b1064070 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 10 client.17628 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a8b14257e0 con 0x55a8b1064070 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe80043d0 con 0x55a8b1064070 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f3fec00f040 con 0x55a8b1064070 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 10 client.17628 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 10 client.17628 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 20 client.17628 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b10229f0 0x55a8b1428d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 4 client.17628 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8b1383df0 con 0x55a8b10229f0 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.542+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b10229f0 0x55a8b1428d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.543+0000 7f3fd3fff640 10 client.17628 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.543+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b10229f0 0x55a8b1428d10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b7913f1763480cc7 server_cookie=9db9ea5f3aaf0b70 in_seq=0 out_seq=0 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1505142636 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a8b1383df0 con 0x55a8b10229f0 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f3fd3fff640 10 client.17628 handle_command_reply: tid=0 2024-09-21T05:56:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 client.17628 shutdown 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 msgr2=0x55a8b1422d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0aa930 tx=0x7f3fec0057e0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 shutdown_connections 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b10229f0 0x55a8b1428d10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b1064070 0x55a8b1422d90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f3fd3fff640 0 client.17628 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b1416ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 --2- 172.21.15.28:0/1505142636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b103b1b0 0x55a8b14169b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 >> 172.21.15.28:0/1505142636 conn(0x55a8b1058460 msgr2=0x55a8b140e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 shutdown_connections 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 1 -- 172.21.15.28:0/1505142636 wait complete. 2024-09-21T05:56:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.544+0000 7f40016fa740 20 client.17628 trim_cache size 0 max 16384 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 Processor -- start 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 -- start start 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1043390 0x55a8b141f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1064070 0x55a8b141f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41af760 con 0x55a8b103b1b0 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41b1360 con 0x55a8b1043390 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f40016fa740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41afe60 con 0x55a8b1064070 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47228/0 (socket says 172.21.15.28:47228) 2024-09-21T05:56:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1864463338 learned_addr learned my addr 172.21.15.28:0/1864463338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1043390 0x55a8b141f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff9723640 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1064070 0x55a8b141f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1864463338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1064070 msgr2=0x55a8b141f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1064070 0x55a8b141f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.558+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1864463338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1043390 msgr2=0x55a8b141f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1043390 0x55a8b141f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1864463338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b140d5b0 con 0x55a8b103b1b0 2024-09-21T05:56:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3fe402a930 tx=0x7f3fe403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0d52d7c14043fb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3fda7fc640 1 -- 172.21.15.28:0/1864463338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4058070 con 0x55a8b103b1b0 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3fda7fc640 1 -- 172.21.15.28:0/1864463338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe4005040 con 0x55a8b103b1b0 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.559+0000 7f3fda7fc640 1 -- 172.21.15.28:0/1864463338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4035040 con 0x55a8b103b1b0 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- 172.21.15.28:0/1864463338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 msgr2=0x55a8b1416e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3fe402a930 tx=0x7f3fe403f000 comp rx=0 tx=0).stop 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- 172.21.15.28:0/1864463338 shutdown_connections 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1064070 0x55a8b141f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1043390 0x55a8b141f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 --2- 172.21.15.28:0/1864463338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b1416e70 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- 172.21.15.28:0/1864463338 >> 172.21.15.28:0/1864463338 conn(0x55a8b1058460 msgr2=0x55a8b1416760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- 172.21.15.28:0/1864463338 shutdown_connections 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- 172.21.15.28:0/1864463338 wait complete. 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 -- start start 2024-09-21T05:56:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.560+0000 7f40016fa740 1 Processor -- start 2024-09-21T05:56:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b148f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b148d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41b0c60 con 0x55a8b103b1b0 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41b0560 con 0x55a8b1064070 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff41ad740 con 0x55a8b1043390 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f40016fa740 10 client.0 _ll_register_callbacks cb 0x7f3fec0a9d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47234/0 (socket says 172.21.15.28:47234) 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1496120085 learned_addr learned my addr 172.21.15.28:0/1496120085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b148f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff9723640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b148d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 msgr2=0x55a8b148f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b148f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1496120085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 msgr2=0x55a8b148d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b148d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff8f22640 1 -- 172.21.15.28:0/1496120085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b148df50 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.561+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b148f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3ff8f22640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40317f0 tx=0x7f3fe4031820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5af75e0ee0d914e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4061060 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f3fe4004340 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe4005040 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3fe4058040 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 10 client.17634 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a8b1498580 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe4005040 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f3fe40035f0 con 0x55a8b103b1b0 2024-09-21T05:56:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 10 client.17634 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 10 client.17634 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 20 client.17634 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b0ff7da0 0x55a8b149acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 4 client.17634 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.562+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8b137f8a0 con 0x55a8b0ff7da0 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.563+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b0ff7da0 0x55a8b149acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.563+0000 7f3fd3fff640 10 client.17634 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.563+0000 7f3ff3fff640 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b0ff7da0 0x55a8b149acf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb5062fcc7e6e176 server_cookie=4004f67e0b9302b0 in_seq=0 out_seq=0 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f3fd3fff640 1 -- 172.21.15.28:0/1496120085 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8b137f8a0 con 0x55a8b0ff7da0 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f3fd3fff640 10 client.17634 handle_command_reply: tid=0 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 client.17634 shutdown 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 msgr2=0x55a8b148eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40317f0 tx=0x7f3fe4031820 comp rx=0 tx=0).stop 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 shutdown_connections 2024-09-21T05:56:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x55a8b0ff7da0 0x55a8b149acf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f3fd3fff640 0 client.17634 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a8b1064070 0x55a8b148d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a8b1043390 0x55a8b148f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 --2- 172.21.15.28:0/1496120085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8b103b1b0 0x55a8b148eb30 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 >> 172.21.15.28:0/1496120085 conn(0x55a8b1058460 msgr2=0x55a8b140e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 shutdown_connections 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 1 -- 172.21.15.28:0/1496120085 wait complete. 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.564+0000 7f40016fa740 20 client.17634 trim_cache size 0 max 16384 2024-09-21T05:56:01.570 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 msgr2=0x7f3fdc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 0x7f3fdc0b8b90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4000c00 tx=0x7f3fe4031400 comp rx=0 tx=0).stop 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 msgr2=0x7f3ff419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00eaa0 tx=0x7f3fec00ef60 comp rx=0 tx=0).stop 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 shutdown_connections 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3fdc0b6680 0x7f3fdc0b8b90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ff4108d20 0x7f3ff419bd60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ff41083d0 0x7f3ff419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 --2- 172.21.15.28:0/2193642216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ff4102ae0 0x7f3ff419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 >> 172.21.15.28:0/2193642216 conn(0x7f3ff4079760 msgr2=0x7f3ff41015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 shutdown_connections 2024-09-21T05:56:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:01.565+0000 7f3ffb186640 1 -- 172.21.15.28:0/2193642216 wait complete. 2024-09-21T05:56:02.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:56:02.537 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:02 smithi028 ceph-mon[31227]: pgmap v599: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-09-21T05:56:02.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:02 smithi129 ceph-mon[34885]: pgmap v599: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-09-21T05:56:02.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:02 smithi181 ceph-mon[34819]: pgmap v599: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 -- 172.21.15.28:0/4225741703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8109df0 msgr2=0x7fe1d8110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 --2- 172.21.15.28:0/4225741703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8109df0 0x7fe1d8110620 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009ca0 tx=0x7fe1c802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 -- 172.21.15.28:0/4225741703 shutdown_connections 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 --2- 172.21.15.28:0/4225741703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8109df0 0x7fe1d8110620 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 --2- 172.21.15.28:0/4225741703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 0x7fe1d81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 --2- 172.21.15.28:0/4225741703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8103b60 0x7fe1d8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 -- 172.21.15.28:0/4225741703 >> 172.21.15.28:0/4225741703 conn(0x7fe1d807b850 msgr2=0x7fe1d807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 -- 172.21.15.28:0/4225741703 shutdown_connections 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.998+0000 7fe1df776640 1 -- 172.21.15.28:0/4225741703 wait complete. 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 Processor -- start 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 -- start start 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 0x7fe1d819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 0x7fe1d819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d8113b60 con 0x7fe1d8103b60 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81139e0 con 0x7fe1d81094a0 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1df776640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d8113ce0 con 0x7fe1d8109df0 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1dd512640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1dd512640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47242/0 (socket says 172.21.15.28:47242) 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1dd512640 1 -- 172.21.15.28:0/2307790347 learned_addr learned my addr 172.21.15.28:0/2307790347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:02.999+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 0x7fe1d819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 -- 172.21.15.28:0/2307790347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 msgr2=0x7fe1d819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 0x7fe1d819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 0x7fe1d819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 -- 172.21.15.28:0/2307790347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 msgr2=0x7fe1d819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 0x7fe1d819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 -- 172.21.15.28:0/2307790347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d81a0420 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 0x7fe1d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1dd512640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b4a0 tx=0x7fe1d000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1d67fc640 1 -- 172.21.15.28:0/2307790347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0014070 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d81a0710 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1d67fc640 1 -- 172.21.15.28:0/2307790347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d0010ac0 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1d67fc640 1 -- 172.21.15.28:0/2307790347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0017c30 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.000+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d81a6e70 con 0x7fe1d8103b60 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 Processor -- start 2024-09-21T05:56:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 -- start start 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1d67fc640 1 -- 172.21.15.28:0/2307790347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe1d0017070 con 0x7fe1d8103b60 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eeca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eecb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df2ee40ea0 con 0x55df2eafa1b0 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df2ee302d0 con 0x55df2eb02390 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55df2ee340e0 con 0x55df2eb23070 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1dd512640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1dd512640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47254/0 (socket says 172.21.15.28:47254) 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.003+0000 7fe1dd512640 1 -- 172.21.15.28:0/1823977995 learned_addr learned my addr 172.21.15.28:0/1823977995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eecb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eeca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 -- 172.21.15.28:0/1823977995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 msgr2=0x55df2eeca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eeca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 -- 172.21.15.28:0/1823977995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 msgr2=0x55df2eecb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eecb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 -- 172.21.15.28:0/1823977995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df2eb8e650 con 0x55df2eafa1b0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eecb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1dd512640 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00e43e0 tx=0x7fe1d00eaf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce39dd91393b5b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1bf7fe640 1 -- 172.21.15.28:0/1823977995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00fc070 con 0x55df2eafa1b0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1bf7fe640 1 -- 172.21.15.28:0/1823977995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d00e8070 con 0x55df2eafa1b0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1bf7fe640 1 -- 172.21.15.28:0/1823977995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00f4570 con 0x55df2eafa1b0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.004+0000 7fe1d67fc640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 0x7fe1c00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 0x7fe1c00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1d67fc640 1 -- 172.21.15.28:0/2307790347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fe1d00e6d50 con 0x7fe1d8103b60 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 -- 172.21.15.28:0/1823977995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 msgr2=0x55df2eecabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 0x7fe1c00b8b90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006010 tx=0x7fe1c8004a40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00e43e0 tx=0x7fe1d00eaf40 comp rx=0 tx=0).stop 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 -- 172.21.15.28:0/1823977995 shutdown_connections 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eecb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eafa1b0 0x55df2eecabe0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/1823977995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eeca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.005+0000 7fe1e5cea740 1 -- 172.21.15.28:0/1823977995 >> 172.21.15.28:0/1823977995 conn(0x55df2eb17460 msgr2=0x55df2eec8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.006+0000 7fe1e5cea740 1 -- 172.21.15.28:0/1823977995 shutdown_connections 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.006+0000 7fe1e5cea740 1 -- 172.21.15.28:0/1823977995 wait complete. 2024-09-21T05:56:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.006+0000 7fe1e5cea740 1 -- start start 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.006+0000 7fe1e5cea740 1 Processor -- start 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eed6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df2ec91700 con 0x55df2eb23070 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df2edcb160 con 0x55df2eb02390 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55df2eed8e80 con 0x55df2eafa1b0 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1e5cea740 10 client.0 _ll_register_callbacks cb 0x55df2ede49e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47270/0 (socket says 172.21.15.28:47270) 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 -- 172.21.15.28:0/118098239 learned_addr learned my addr 172.21.15.28:0/118098239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1dd512640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eed6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 -- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 msgr2=0x55df2eed5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 -- 172.21.15.28:0/118098239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 msgr2=0x55df2eed6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eed6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 -- 172.21.15.28:0/118098239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df2edf8c90 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1dd512640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eed6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8035000 tx=0x7fe1c802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577712e1b7167ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.007+0000 7fe1bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8004250 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fe1c80043f0 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1d00e8070 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe1c80045f0 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 10 client.17646 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55df2eee4bf0 con 0x55df2eb23070 2024-09-21T05:56:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d00e8070 con 0x55df2eb23070 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7fe1c8050070 con 0x55df2eb23070 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 10 client.17646 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 10 client.17646 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 20 client.17646 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eae19f0 0x55df2eee80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 4 client.17646 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.008+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df2ee42df0 con 0x55df2eae19f0 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.009+0000 7fe1dd512640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eae19f0 0x55df2eee80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.009+0000 7fe1dd512640 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eae19f0 0x55df2eee80e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c1268a368e086b server_cookie=f94a337b07cea466 in_seq=0 out_seq=0 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.009+0000 7fe1bcff9640 10 client.17646 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1bcff9640 1 -- 172.21.15.28:0/118098239 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55df2ee42df0 con 0x55df2eae19f0 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1bcff9640 10 client.17646 handle_command_reply: tid=0 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 client.17646 shutdown 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 msgr2=0x55df2eee21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8035000 tx=0x7fe1c802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 shutdown_connections 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eae19f0 0x55df2eee80e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb23070 0x55df2eee21a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb02390 0x55df2eed6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/118098239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 >> 172.21.15.28:0/118098239 conn(0x55df2eb17460 msgr2=0x55df2eecd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1bcff9640 0 client.17646 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 shutdown_connections 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.010+0000 7fe1e5cea740 1 -- 172.21.15.28:0/118098239 wait complete. 2024-09-21T05:56:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.011+0000 7fe1e5cea740 20 client.17646 trim_cache size 0 max 16384 2024-09-21T05:56:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 Processor -- start 2024-09-21T05:56:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 -- start start 2024-09-21T05:56:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 0x55df2eec85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eed2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81af8d0 con 0x55df2eb02390 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81b14d0 con 0x55df2eafa1b0 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81affd0 con 0x55df2eb23070 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47282/0 (socket says 172.21.15.28:47282) 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 -- 172.21.15.28:0/2902939474 learned_addr learned my addr 172.21.15.28:0/2902939474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eed2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dd512640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 0x55df2eec85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 -- 172.21.15.28:0/2902939474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 msgr2=0x55df2eed2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eed2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 -- 172.21.15.28:0/2902939474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 msgr2=0x55df2eec85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 0x55df2eec85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dcd11640 1 -- 172.21.15.28:0/2902939474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df2eed70a0 con 0x55df2eb02390 2024-09-21T05:56:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.024+0000 7fe1dd512640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 0x55df2eec85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.025+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00b980 tx=0x7fe1cc00be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e552503b68ed475 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.025+0000 7fe1beffd640 1 -- 172.21.15.28:0/2902939474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc003db0 con 0x55df2eb02390 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.025+0000 7fe1beffd640 1 -- 172.21.15.28:0/2902939474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1cc020070 con 0x55df2eb02390 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.025+0000 7fe1beffd640 1 -- 172.21.15.28:0/2902939474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc01b660 con 0x55df2eb02390 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- 172.21.15.28:0/2902939474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 msgr2=0x55df2eee20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00b980 tx=0x7fe1cc00be40 comp rx=0 tx=0).stop 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- 172.21.15.28:0/2902939474 shutdown_connections 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eb23070 0x55df2eed2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eee20a0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/2902939474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eafa1b0 0x55df2eec85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- 172.21.15.28:0/2902939474 >> 172.21.15.28:0/2902939474 conn(0x55df2eb17460 msgr2=0x55df2eec7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- 172.21.15.28:0/2902939474 shutdown_connections 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- 172.21.15.28:0/2902939474 wait complete. 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 -- start start 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.026+0000 7fe1e5cea740 1 Processor -- start 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb23070 0x55df2ef4aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81b0dd0 con 0x55df2eb02390 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81b06d0 con 0x55df2eb23070 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81ad8b0 con 0x55df2eafa1b0 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1e5cea740 10 client.0 _ll_register_callbacks cb 0x7fe1c802f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47292/0 (socket says 172.21.15.28:47292) 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 -- 172.21.15.28:0/542866124 learned_addr learned my addr 172.21.15.28:0/542866124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dd512640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1ddd13640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb23070 0x55df2ef4aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 -- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 msgr2=0x55df2eed9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 -- 172.21.15.28:0/542866124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb23070 msgr2=0x55df2ef4aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb23070 0x55df2ef4aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.027+0000 7fe1dcd11640 1 -- 172.21.15.28:0/542866124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df2ef4b0d0 con 0x55df2eb02390 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1dcd11640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc011ea0 tx=0x7fe1cc0116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd536d40eb08a72e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc011ae0 con 0x55df2eb02390 2024-09-21T05:56:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fe1cc025400 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1cc020070 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe1cc004340 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 10 client.17652 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55df2ef52340 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1cc020070 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7fe1cc02e640 con 0x55df2eb02390 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 10 client.17652 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 10 client.17652 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 20 client.17652 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eab6da0 0x55df2ef54b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 4 client.17652 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.028+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df2ee3e8a0 con 0x55df2eab6da0 2024-09-21T05:56:03.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.029+0000 7fe1dd512640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eab6da0 0x55df2ef54b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:03.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.029+0000 7fe1dd512640 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eab6da0 0x55df2ef54b20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0018eca4c87f9d4 server_cookie=a4e7c1e6433974d3 in_seq=0 out_seq=0 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.029+0000 7fe1bcff9640 10 client.17652 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1bcff9640 1 -- 172.21.15.28:0/542866124 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df2ee3e8a0 con 0x55df2eab6da0 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1bcff9640 10 client.17652 handle_command_reply: tid=0 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 client.17652 shutdown 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 msgr2=0x55df2eed9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc011ea0 tx=0x7fe1cc0116f0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 shutdown_connections 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x55df2eab6da0 0x55df2ef54b20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df2eb23070 0x55df2ef4aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1bcff9640 0 client.17652 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df2eb02390 0x55df2eed9f50 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 --2- 172.21.15.28:0/542866124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55df2eafa1b0 0x55df2eed9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 >> 172.21.15.28:0/542866124 conn(0x55df2eb17460 msgr2=0x55df2eecb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 shutdown_connections 2024-09-21T05:56:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 1 -- 172.21.15.28:0/542866124 wait complete. 2024-09-21T05:56:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.030+0000 7fe1e5cea740 20 client.17652 trim_cache size 0 max 16384 2024-09-21T05:56:03.037 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 msgr2=0x7fe1c00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 0x7fe1c00b8b90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006010 tx=0x7fe1c8004a40 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 msgr2=0x7fe1d819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b4a0 tx=0x7fe1d000b960 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 shutdown_connections 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d8109df0 0x7fe1d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1d81094a0 0x7fe1d819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe1c00b6680 0x7fe1c00b8b90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 --2- 172.21.15.28:0/2307790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1d8103b60 0x7fe1d819a4b0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 >> 172.21.15.28:0/2307790347 conn(0x7fe1d807b850 msgr2=0x7fe1d81074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 shutdown_connections 2024-09-21T05:56:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:03.031+0000 7fe1df776640 1 -- 172.21.15.28:0/2307790347 wait complete. 2024-09-21T05:56:03.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:56:03.994 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:04.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:04 smithi028 ceph-mon[31227]: pgmap v600: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-21T05:56:04.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 -- 172.21.15.28:0/2160400198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc41083d0 msgr2=0x7fdfc41087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2160400198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc41083d0 0x7fdfc41087e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009900 tx=0x7fdfac02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:04.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 -- 172.21.15.28:0/2160400198 shutdown_connections 2024-09-21T05:56:04.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2160400198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc4108d20 0x7fdfc410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2160400198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc41083d0 0x7fdfc41087e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2160400198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 -- 172.21.15.28:0/2160400198 >> 172.21.15.28:0/2160400198 conn(0x7fdfc4079760 msgr2=0x7fdfc4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 -- 172.21.15.28:0/2160400198 shutdown_connections 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.422+0000 7fdfca62e640 1 -- 172.21.15.28:0/2160400198 wait complete. 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.423+0000 7fdfca62e640 1 Processor -- start 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.423+0000 7fdfca62e640 1 -- start start 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.423+0000 7fdfca62e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.423+0000 7fdfca62e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41083d0 0x7fdfc419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfca62e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfca62e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4113ab0 con 0x7fdfc4108d20 2024-09-21T05:56:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfca62e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4113930 con 0x7fdfc4102ae0 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfca62e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4113c30 con 0x7fdfc41083d0 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47302/0 (socket says 172.21.15.28:47302) 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 -- 172.21.15.28:0/2365515663 learned_addr learned my addr 172.21.15.28:0/2365515663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41083d0 0x7fdfc419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 -- 172.21.15.28:0/2365515663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41083d0 msgr2=0x7fdfc419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41083d0 0x7fdfc419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 -- 172.21.15.28:0/2365515663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 msgr2=0x7fdfc419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 -- 172.21.15.28:0/2365515663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc41a0470 con 0x7fdfc4108d20 2024-09-21T05:56:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00cad0 tx=0x7fdfbc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.424+0000 7fdfc1ffb640 1 -- 172.21.15.28:0/2365515663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00ae20 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.425+0000 7fdfc1ffb640 1 -- 172.21.15.28:0/2365515663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfbc005020 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.425+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc41a0760 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.425+0000 7fdfc1ffb640 1 -- 172.21.15.28:0/2365515663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00f6b0 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.425+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc41a6ec0 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.427+0000 7fdfd0ba2740 1 Processor -- start 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.427+0000 7fdfc1ffb640 1 -- 172.21.15.28:0/2365515663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fdfbc004b20 con 0x7fdfc4108d20 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 -- start start 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4bdfd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4bdfe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf4bd73ea0 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf4bd632d0 con 0x55cf4ba56070 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cf4bd670e0 con 0x55cf4ba35390 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47310/0 (socket says 172.21.15.28:47310) 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfc3fff640 1 -- 172.21.15.28:0/520828012 learned_addr learned my addr 172.21.15.28:0/520828012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4bdfd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4bdfe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.428+0000 7fdfc3fff640 1 -- 172.21.15.28:0/520828012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 msgr2=0x55cf4bdfe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4bdfe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc3fff640 1 -- 172.21.15.28:0/520828012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 msgr2=0x55cf4bdfd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4bdfd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc3fff640 1 -- 172.21.15.28:0/520828012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf4bac1650 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4bdfd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc1ffb640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 0x7fdfb00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 0x7fdfb00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400afd0 tx=0x7fdfb400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d9399241441c35 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfc1ffb640 1 -- 172.21.15.28:0/2365515663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fdfbc0e7650 con 0x7fdfc4108d20 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfba7fc640 1 -- 172.21.15.28:0/520828012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb401d070 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfba7fc640 1 -- 172.21.15.28:0/520828012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb4010db0 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.429+0000 7fdfba7fc640 1 -- 172.21.15.28:0/520828012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4018e30 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/520828012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 msgr2=0x55cf4bdfdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400afd0 tx=0x7fdfb400f850 comp rx=0 tx=0).stop 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 0x7fdfb00b8b90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdfac000c00 tx=0x7fdfac005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/520828012 shutdown_connections 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4bdfe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfdbe0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/520828012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4bdfd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/520828012 >> 172.21.15.28:0/520828012 conn(0x55cf4ba4a460 msgr2=0x55cf4bdfb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/520828012 shutdown_connections 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.430+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/520828012 wait complete. 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.431+0000 7fdfd0ba2740 1 -- start start 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.431+0000 7fdfd0ba2740 1 Processor -- start 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be08cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be09770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf4bbc4700 con 0x55cf4ba35390 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf4bcfe160 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cf4be0bf20 con 0x55cf4ba56070 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfd0ba2740 10 client.0 _ll_register_callbacks cb 0x55cf4bd179e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47324/0 (socket says 172.21.15.28:47324) 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3181236752 learned_addr learned my addr 172.21.15.28:0/3181236752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be09770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be08cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3181236752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 msgr2=0x55cf4be09770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be09770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 msgr2=0x55cf4be08cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be08cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfbbfff640 1 -- 172.21.15.28:0/3181236752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf4bd2bc90 con 0x55cf4ba35390 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.432+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be08cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009900 tx=0x7fdfac031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d294fb731c7aa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac04f020 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fdfac003a00 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfb4010db0 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdfac035040 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfd0ba2740 10 client.17664 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cf4be17ba0 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb4010db0 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7fdfac03a040 con 0x55cf4ba35390 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfd0ba2740 10 client.17664 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.433+0000 7fdfd0ba2740 10 client.17664 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfd0ba2740 20 client.17664 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:04.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4ba149f0 0x55cf4be1b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfd0ba2740 4 client.17664 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf4bd75df0 con 0x55cf4ba149f0 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4ba149f0 0x55cf4be1b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4ba149f0 0x55cf4be1b1c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b810bd6dc5cbe9a server_cookie=c4f41d3a59887b47 in_seq=0 out_seq=0 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.434+0000 7fdfa37fe640 10 client.17664 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.435+0000 7fdfa37fe640 1 -- 172.21.15.28:0/3181236752 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cf4bd75df0 con 0x55cf4ba149f0 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.435+0000 7fdfa37fe640 10 client.17664 handle_command_reply: tid=0 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.435+0000 7fdfd0ba2740 1 client.17664 shutdown 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.435+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 msgr2=0x55cf4be09230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.435+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009900 tx=0x7fdfac031400 comp rx=0 tx=0).stop 2024-09-21T05:56:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 shutdown_connections 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4ba149f0 0x55cf4be1b1c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be09770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be09230 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfa37fe640 0 client.17664 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3181236752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be08cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 >> 172.21.15.28:0/3181236752 conn(0x55cf4ba4a460 msgr2=0x55cf4be00830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 shutdown_connections 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3181236752 wait complete. 2024-09-21T05:56:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.436+0000 7fdfd0ba2740 20 client.17664 trim_cache size 0 max 16384 2024-09-21T05:56:04.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.449+0000 7fdfd0ba2740 1 Processor -- start 2024-09-21T05:56:04.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.449+0000 7fdfd0ba2740 1 -- start start 2024-09-21T05:56:04.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.449+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4be05ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4be06420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41af780 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b1380 con 0x55cf4ba56070 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41afe80 con 0x55cf4ba35390 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47334/0 (socket says 172.21.15.28:47334) 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 -- 172.21.15.28:0/3092395960 learned_addr learned my addr 172.21.15.28:0/3092395960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4be06420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4be05ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 -- 172.21.15.28:0/3092395960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 msgr2=0x55cf4be05ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4be05ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 -- 172.21.15.28:0/3092395960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 msgr2=0x55cf4be06420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4be06420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 -- 172.21.15.28:0/3092395960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf4bdfc350 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400fd50 tx=0x7fdfb4005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc789ba3b284b7ec server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfb9ffb640 1 -- 172.21.15.28:0/3092395960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb401d040 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.450+0000 7fdfb9ffb640 1 -- 172.21.15.28:0/3092395960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb401e040 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfb9ffb640 1 -- 172.21.15.28:0/3092395960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40035f0 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3092395960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 msgr2=0x55cf4bdfede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400fd50 tx=0x7fdfb4005690 comp rx=0 tx=0).stop 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3092395960 shutdown_connections 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba56070 0x55cf4be06420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba35390 0x55cf4be05ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/3092395960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba2d1b0 0x55cf4bdfede0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3092395960 >> 172.21.15.28:0/3092395960 conn(0x55cf4ba4a460 msgr2=0x55cf4bdfe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3092395960 shutdown_connections 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.451+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/3092395960 wait complete. 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 -- start start 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 Processor -- start 2024-09-21T05:56:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be0d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be0dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b0c80 con 0x55cf4ba35390 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41b0580 con 0x55cf4ba2d1b0 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41ad760 con 0x55cf4ba56070 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfd0ba2740 10 client.0 _ll_register_callbacks cb 0x7fdfac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfbbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfbbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47346/0 (socket says 172.21.15.28:47346) 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.452+0000 7fdfbbfff640 1 -- 172.21.15.28:0/4003730493 learned_addr learned my addr 172.21.15.28:0/4003730493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be0d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfc8bcb640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be0dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 -- 172.21.15.28:0/4003730493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 msgr2=0x55cf4be0dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be0dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 -- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 msgr2=0x55cf4be0d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be0d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 -- 172.21.15.28:0/4003730493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf4be84c00 con 0x55cf4ba35390 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfbbfff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fdfac04bd40 tx=0x7fdfac003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6f3396d38be194 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfa37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac063060 con 0x55cf4ba35390 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7fdfac04b830 con 0x55cf4ba35390 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfb401e040 con 0x55cf4ba35390 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdfac04f040 con 0x55cf4ba35390 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfd0ba2740 10 client.17670 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.453+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cf4be85e80 con 0x55cf4ba35390 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb401e040 con 0x55cf4ba35390 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7fdfac03ac90 con 0x55cf4ba35390 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 10 client.17670 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 10 client.17670 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 20 client.17670 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4b9e9da0 0x55cf4be88640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 4 client.17670 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf4bd718a0 con 0x55cf4b9e9da0 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.454+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4b9e9da0 0x55cf4be88640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfc3fff640 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4b9e9da0 0x55cf4be88640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea5d9fbba7d0e13c server_cookie=42dc125671009a09 in_seq=0 out_seq=0 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfa37fe640 10 client.17670 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfa37fe640 1 -- 172.21.15.28:0/4003730493 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf4bd718a0 con 0x55cf4b9e9da0 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfa37fe640 10 client.17670 handle_command_reply: tid=0 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 client.17670 shutdown 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 msgr2=0x55cf4be0d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fdfac04bd40 tx=0x7fdfac003ee0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 shutdown_connections 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x55cf4b9e9da0 0x55cf4be88640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf4ba56070 0x55cf4be0dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfa37fe640 0 client.17670 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf4ba35390 0x55cf4be0d740 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 --2- 172.21.15.28:0/4003730493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf4ba2d1b0 0x55cf4be0d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.455+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 >> 172.21.15.28:0/4003730493 conn(0x55cf4ba4a460 msgr2=0x55cf4be14af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 shutdown_connections 2024-09-21T05:56:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfd0ba2740 1 -- 172.21.15.28:0/4003730493 wait complete. 2024-09-21T05:56:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfd0ba2740 20 client.17670 trim_cache size 0 max 16384 2024-09-21T05:56:04.463 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 msgr2=0x7fdfb00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 0x7fdfb00b8b90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdfac000c00 tx=0x7fdfac005a20 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 msgr2=0x7fdfc419fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00cad0 tx=0x7fdfbc00cf90 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 shutdown_connections 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fdfb00b6680 0x7fdfb00b8b90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4108d20 0x7fdfc419fd40 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfc41083d0 0x7fdfc419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 --2- 172.21.15.28:0/2365515663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfc4102ae0 0x7fdfc419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 >> 172.21.15.28:0/2365515663 conn(0x7fdfc4079760 msgr2=0x7fdfc41052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 shutdown_connections 2024-09-21T05:56:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:04.456+0000 7fdfca62e640 1 -- 172.21.15.28:0/2365515663 wait complete. 2024-09-21T05:56:04.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:04 smithi129 ceph-mon[34885]: pgmap v600: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-21T05:56:04.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:04 smithi181 ceph-mon[34819]: pgmap v600: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-09-21T05:56:05.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:56:05.455 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 -- 172.21.15.28:0/231077194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201083a0 msgr2=0x7f91201087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 --2- 172.21.15.28:0/231077194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201083a0 0x7f91201087b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9114009900 tx=0x7f911402a410 comp rx=0 tx=0).stop 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 -- 172.21.15.28:0/231077194 shutdown_connections 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 --2- 172.21.15.28:0/231077194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9120108cf0 0x7f912010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 --2- 172.21.15.28:0/231077194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201083a0 0x7f91201087b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 --2- 172.21.15.28:0/231077194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 0x7f9120102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 -- 172.21.15.28:0/231077194 >> 172.21.15.28:0/231077194 conn(0x7f9120079730 msgr2=0x7f9120079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 -- 172.21.15.28:0/231077194 shutdown_connections 2024-09-21T05:56:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.948+0000 7f9128519640 1 -- 172.21.15.28:0/231077194 wait complete. 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.949+0000 7f9128519640 1 Processor -- start 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.949+0000 7f9128519640 1 -- start start 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 0x7f912019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91201083a0 0x7f912019b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9120112960 con 0x7f9120108cf0 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91201127e0 con 0x7f91201083a0 2024-09-21T05:56:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9120112ae0 con 0x7f9120102a60 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39296/0 (socket says 172.21.15.28:39296) 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 -- 172.21.15.28:0/98397042 learned_addr learned my addr 172.21.15.28:0/98397042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9125ab4640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91201083a0 0x7f912019b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f91262b5640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 0x7f912019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 -- 172.21.15.28:0/98397042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 msgr2=0x7f912019b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 0x7f912019b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 -- 172.21.15.28:0/98397042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91201083a0 msgr2=0x7f912019b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91201083a0 0x7f912019b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 -- 172.21.15.28:0/98397042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91201a65c0 con 0x7f9120108cf0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9126ab6640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f911f7fe640 1 -- 172.21.15.28:0/98397042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118017030 con 0x7f9120108cf0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f911f7fe640 1 -- 172.21.15.28:0/98397042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9118004020 con 0x7f9120108cf0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.950+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91201a68b0 con 0x7f9120108cf0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.951+0000 7f911f7fe640 1 -- 172.21.15.28:0/98397042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118003b50 con 0x7f9120108cf0 2024-09-21T05:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.951+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91201a6dc0 con 0x7f9120108cf0 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.953+0000 7f911f7fe640 1 -- 172.21.15.28:0/98397042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f911801a070 con 0x7f9120108cf0 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.953+0000 7f912ea8d740 1 Processor -- start 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 -- start start 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f97b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f97bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55962f8f1ea0 con 0x55962f5d4070 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55962f8e12d0 con 0x55962f5ab1b0 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f912ea8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55962f8e50e0 con 0x55962f5b3390 2024-09-21T05:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39310/0 (socket says 172.21.15.28:39310) 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 -- 172.21.15.28:0/1913722097 learned_addr learned my addr 172.21.15.28:0/1913722097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f91262b5640 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f97b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f97bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 -- 172.21.15.28:0/1913722097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 msgr2=0x55962f97bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f97bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 -- 172.21.15.28:0/1913722097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 msgr2=0x55962f97b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f97b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.954+0000 7f9125ab4640 1 -- 172.21.15.28:0/1913722097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55962f63f650 con 0x55962f5d4070 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f911f7fe640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 0x7f91080b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9114005f70 tx=0x7f9114005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7bcbc3d50ef3ff server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f91262b5640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 0x7f91080b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f911cff9640 1 -- 172.21.15.28:0/1913722097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114044020 con 0x55962f5d4070 2024-09-21T05:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f911f7fe640 1 -- 172.21.15.28:0/98397042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f91180e6df0 con 0x7f9120108cf0 2024-09-21T05:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f911cff9640 1 -- 172.21.15.28:0/1913722097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91140317a0 con 0x55962f5d4070 2024-09-21T05:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.955+0000 7f911cff9640 1 -- 172.21.15.28:0/1913722097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114035040 con 0x55962f5d4070 2024-09-21T05:56:05.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- 172.21.15.28:0/1913722097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 msgr2=0x55962f97b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f91262b5640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 0x7f91080b8c40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55962f979f40 tx=0x7f91100097e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9114005f70 tx=0x7f9114005e50 comp rx=0 tx=0).stop 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- 172.21.15.28:0/1913722097 shutdown_connections 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f97bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f97b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1913722097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f97b2c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- 172.21.15.28:0/1913722097 >> 172.21.15.28:0/1913722097 conn(0x55962f5c8460 msgr2=0x55962f9791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- 172.21.15.28:0/1913722097 shutdown_connections 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- 172.21.15.28:0/1913722097 wait complete. 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.956+0000 7f912ea8d740 1 -- start start 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 Processor -- start 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f98b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5d4070 0x55962f9970d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55962f742700 con 0x55962f5ab1b0 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55962f87c160 con 0x55962f5d4070 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55962f98dda0 con 0x55962f5b3390 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f912ea8d740 10 client.0 _ll_register_callbacks cb 0x55962f8959e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39314/0 (socket says 172.21.15.28:39314) 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 -- 172.21.15.28:0/824163609 learned_addr learned my addr 172.21.15.28:0/824163609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f9126ab6640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5d4070 0x55962f9970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 -- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 msgr2=0x55962f98b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f98b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 -- 172.21.15.28:0/824163609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5d4070 msgr2=0x55962f9970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5d4070 0x55962f9970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.957+0000 7f91262b5640 1 -- 172.21.15.28:0/824163609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55962f8a9c90 con 0x55962f5ab1b0 2024-09-21T05:56:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f91262b5640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f91100109d0 tx=0x7f9110004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be0b2ef2a84b664 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911001db10 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f911001dcb0 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91140317a0 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f911002e460 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 10 client.17682 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55962f999bf0 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91140317a0 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f9110019040 con 0x55962f5ab1b0 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 10 client.17682 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 20 client.17682 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.958+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f5929f0 0x55962f99d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.959+0000 7f912ea8d740 4 client.17682 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.959+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55962f8f3df0 con 0x55962f5929f0 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.959+0000 7f9125ab4640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f5929f0 0x55962f99d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.959+0000 7f9105ffb640 10 client.17682 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.959+0000 7f9125ab4640 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f5929f0 0x55962f99d2d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e95eb973ee4bb90c server_cookie=8a519ef3bff0e64f in_seq=0 out_seq=0 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f9105ffb640 1 -- 172.21.15.28:0/824163609 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55962f8f3df0 con 0x55962f5929f0 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f9105ffb640 10 client.17682 handle_command_reply: tid=0 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 client.17682 shutdown 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 msgr2=0x55962f98ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f91100109d0 tx=0x7f9110004a40 comp rx=0 tx=0).stop 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 shutdown_connections 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f5929f0 0x55962f99d2d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f9105ffb640 0 client.17682 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5d4070 0x55962f9970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f98b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 --2- 172.21.15.28:0/824163609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5ab1b0 0x55962f98ae60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.960+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 >> 172.21.15.28:0/824163609 conn(0x55962f5c8460 msgr2=0x55962f97ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.961+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 shutdown_connections 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.961+0000 7f912ea8d740 1 -- 172.21.15.28:0/824163609 wait complete. 2024-09-21T05:56:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.961+0000 7f912ea8d740 20 client.17682 trim_cache size 0 max 16384 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 Processor -- start 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 -- start start 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f996fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91201ad4e0 con 0x55962f5d4070 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91201af0e0 con 0x55962f5ab1b0 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f912ea8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f91201adbe0 con 0x55962f5b3390 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f9126ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f9126ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39322/0 (socket says 172.21.15.28:39322) 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f9126ab6640 1 -- 172.21.15.28:0/1736574996 learned_addr learned my addr 172.21.15.28:0/1736574996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f996fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.974+0000 7f91262b5640 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 -- 172.21.15.28:0/1736574996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 msgr2=0x55962f996fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f996fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 -- 172.21.15.28:0/1736574996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 msgr2=0x55962f98b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 -- 172.21.15.28:0/1736574996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55962f98bd90 con 0x55962f5d4070 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9126ab6640 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f91180e45a0 tx=0x7f91180ac3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c12588b009d17a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9107fff640 1 -- 172.21.15.28:0/1736574996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180ad340 con 0x55962f5d4070 2024-09-21T05:56:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9107fff640 1 -- 172.21.15.28:0/1736574996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91180b1720 con 0x55962f5d4070 2024-09-21T05:56:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.975+0000 7f9107fff640 1 -- 172.21.15.28:0/1736574996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180a8590 con 0x55962f5d4070 2024-09-21T05:56:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 -- 172.21.15.28:0/1736574996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 msgr2=0x55962f997510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f91180e45a0 tx=0x7f91180ac3b0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 -- 172.21.15.28:0/1736574996 shutdown_connections 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5d4070 0x55962f997510 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5b3390 0x55962f996fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1736574996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 -- 172.21.15.28:0/1736574996 >> 172.21.15.28:0/1736574996 conn(0x55962f5c8460 msgr2=0x55962f98ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 -- 172.21.15.28:0/1736574996 shutdown_connections 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.976+0000 7f912ea8d740 1 -- 172.21.15.28:0/1736574996 wait complete. 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 -- start start 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 Processor -- start 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5d4070 0x55962fa06c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91201ae9e0 con 0x55962f5b3390 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91201ae2e0 con 0x55962f5ab1b0 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f91201ab4c0 con 0x55962f5d4070 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f912ea8d740 10 client.0 _ll_register_callbacks cb 0x7f9110010650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39334/0 (socket says 172.21.15.28:39334) 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 -- 172.21.15.28:0/1696391886 learned_addr learned my addr 172.21.15.28:0/1696391886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f91262b5640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9126ab6640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5d4070 0x55962fa06c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 -- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5d4070 msgr2=0x55962fa06c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5d4070 0x55962fa06c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 -- 172.21.15.28:0/1696391886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 msgr2=0x55962f98f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.977+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9125ab4640 1 -- 172.21.15.28:0/1696391886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55962fa071d0 con 0x55962f5b3390 2024-09-21T05:56:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f91262b5640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9125ab4640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9114005f70 tx=0x7f911402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fd68b9a9d99470 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9105ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114056060 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8528+0+0 (secure 0 0 0) 0x7f9114004240 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91180b1720 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9114044040 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f912ea8d740 10 client.17688 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55962fa08390 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.978+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91180b1720 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f91140046c0 con 0x55962f5b3390 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 10 client.17688 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 20 client.17688 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f567da0 0x55962fa0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 4 client.17688 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55962f8ef8a0 con 0x55962f567da0 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f91262b5640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f567da0 0x55962fa0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f91262b5640 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f567da0 0x55962fa0a760 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74686a21147f5030 server_cookie=41126610398cf9f8 in_seq=0 out_seq=0 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.979+0000 7f9105ffb640 10 client.17688 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f9105ffb640 1 -- 172.21.15.28:0/1696391886 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55962f8ef8a0 con 0x55962f567da0 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f9105ffb640 10 client.17688 handle_command_reply: tid=0 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 client.17688 shutdown 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 msgr2=0x55962f98fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9114005f70 tx=0x7f911402a410 comp rx=0 tx=0).stop 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 shutdown_connections 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55962f567da0 0x55962fa0a760 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55962f5d4070 0x55962fa06c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f9105ffb640 0 client.17688 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55962f5b3390 0x55962f98fc20 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 --2- 172.21.15.28:0/1696391886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55962f5ab1b0 0x55962f98f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 >> 172.21.15.28:0/1696391886 conn(0x55962f5c8460 msgr2=0x55962f9789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 shutdown_connections 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 1 -- 172.21.15.28:0/1696391886 wait complete. 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.980+0000 7f912ea8d740 20 client.17688 trim_cache size 0 max 16384 2024-09-21T05:56:05.987 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 msgr2=0x7f91080b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 0x7f91080b8c40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55962f979f40 tx=0x7f91100097e0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 msgr2=0x7f912019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800de10 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 shutdown_connections 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91080b6730 0x7f91080b8c40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120108cf0 0x7f912019bdb0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91201083a0 0x7f912019b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 --2- 172.21.15.28:0/98397042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9120102a60 0x7f912019b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 >> 172.21.15.28:0/98397042 conn(0x7f9120079730 msgr2=0x7f91201044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 shutdown_connections 2024-09-21T05:56:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:05.981+0000 7f9128519640 1 -- 172.21.15.28:0/98397042 wait complete. 2024-09-21T05:56:06.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:06 smithi129 ceph-mon[34885]: pgmap v601: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T05:56:06.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:06 smithi028 ceph-mon[31227]: pgmap v601: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T05:56:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:06 smithi181 ceph-mon[34819]: pgmap v601: 97 pgs: 97 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T05:56:07.759 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-21T05:56:07.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-21T05:56:07.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:07 smithi028 ceph-mon[31227]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T05:56:07.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:07 smithi181 ceph-mon[34819]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T05:56:07.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:07 smithi129 ceph-mon[34885]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T05:56:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:08.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:08 smithi028 ceph-mon[31227]: pgmap v603: 97 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 82 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 -- 172.21.15.28:0/3637419299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 msgr2=0x7f806810d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 --2- 172.21.15.28:0/3637419299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806810d850 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8058009900 tx=0x7f805802a410 comp rx=0 tx=0).stop 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 -- 172.21.15.28:0/3637419299 shutdown_connections 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 --2- 172.21.15.28:0/3637419299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f8068110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 --2- 172.21.15.28:0/3637419299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806810d850 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 --2- 172.21.15.28:0/3637419299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f8068104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 -- 172.21.15.28:0/3637419299 >> 172.21.15.28:0/3637419299 conn(0x7f8068079730 msgr2=0x7f8068079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 -- 172.21.15.28:0/3637419299 shutdown_connections 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.556+0000 7f807054c640 1 -- 172.21.15.28:0/3637419299 wait complete. 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.557+0000 7f807054c640 1 Processor -- start 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.557+0000 7f807054c640 1 -- start start 2024-09-21T05:56:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.557+0000 7f807054c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f806819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f807054c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f807054c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f806819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f807054c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8068113a70 con 0x7f8068105300 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f807054c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80681138f0 con 0x7f80681049b0 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f807054c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8068113bf0 con 0x7f806810de10 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f806819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39350/0 (socket says 172.21.15.28:39350) 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 -- 172.21.15.28:0/2865783275 learned_addr learned my addr 172.21.15.28:0/2865783275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806eae9640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f806819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 -- 172.21.15.28:0/2865783275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 msgr2=0x7f806819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f806819fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 -- 172.21.15.28:0/2865783275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 msgr2=0x7f806819a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f806819a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 -- 172.21.15.28:0/2865783275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80681a04b0 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806eae9640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f806819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806e2e8640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f806819a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.558+0000 7f806dae7640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.559+0000 7f805f7fe640 1 -- 172.21.15.28:0/2865783275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805803f030 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.559+0000 7f805f7fe640 1 -- 172.21.15.28:0/2865783275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80580043d0 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.559+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80681a0740 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.559+0000 7f805f7fe640 1 -- 172.21.15.28:0/2865783275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058035070 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.559+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80681a6f00 con 0x7f8068105300 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.562+0000 7f8076ac0740 1 Processor -- start 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.562+0000 7f8076ac0740 1 -- start start 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6b8490 0x55636da88970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da88eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55636d9fee60 con 0x55636d6b8490 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636d9ee290 con 0x55636d6c0670 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f8076ac0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55636d9f20a0 con 0x55636d6e1350 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6b8490 0x55636da88970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 -- 172.21.15.28:0/2316730347 learned_addr learned my addr 172.21.15.28:0/2316730347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f805f7fe640 1 -- 172.21.15.28:0/2865783275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8058042030 con 0x7f8068105300 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 -- 172.21.15.28:0/2316730347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 msgr2=0x55636da88eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da88eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 -- 172.21.15.28:0/2316730347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6b8490 msgr2=0x55636da88970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6b8490 0x55636da88970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.563+0000 7f806eae9640 1 -- 172.21.15.28:0/2316730347 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636d74c930 con 0x55636d6e1350 2024-09-21T05:56:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.564+0000 7f806eae9640 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8060011790 tx=0x7f8060011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b502ac24b0ffc85 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.564+0000 7f805cff9640 1 -- 172.21.15.28:0/2316730347 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060003c30 con 0x55636d6e1350 2024-09-21T05:56:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.565+0000 7f805cff9640 1 -- 172.21.15.28:0/2316730347 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f806000fe50 con 0x55636d6e1350 2024-09-21T05:56:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.565+0000 7f805cff9640 1 -- 172.21.15.28:0/2316730347 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80600193e0 con 0x55636d6e1350 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.565+0000 7f805f7fe640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 0x7f80540b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.565+0000 7f806e2e8640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 0x7f80540b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.565+0000 7f805f7fe640 1 -- 172.21.15.28:0/2865783275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f8058107c00 con 0x7f8068105300 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- 172.21.15.28:0/2316730347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 msgr2=0x55636da88430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8060011790 tx=0x7f8060011c50 comp rx=0 tx=0).stop 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- 172.21.15.28:0/2316730347 shutdown_connections 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da88eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6b8490 0x55636da88970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 --2- 172.21.15.28:0/2316730347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da88430 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- 172.21.15.28:0/2316730347 >> 172.21.15.28:0/2316730347 conn(0x55636d6d5740 msgr2=0x55636da86320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f806e2e8640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 0x7f80540b8c10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55636da870b0 tx=0x7f8064005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- 172.21.15.28:0/2316730347 shutdown_connections 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- 172.21.15.28:0/2316730347 wait complete. 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 -- start start 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.566+0000 7f8076ac0740 1 Processor -- start 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da94590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6e1350 0x55636da9ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55636d84f890 con 0x55636d6e1350 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636d988d40 con 0x55636d6c0670 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55636da96c10 con 0x55636d6b8490 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f8076ac0740 10 client.0 _ll_register_callbacks cb 0x55636d9ecc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:08.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:36804/0 (socket says 172.21.15.28:36804) 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 -- 172.21.15.28:0/3710621780 learned_addr learned my addr 172.21.15.28:0/3710621780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806eae9640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6e1350 0x55636da9ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806dae7640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da94590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 -- 172.21.15.28:0/3710621780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 msgr2=0x55636da94590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da94590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 -- 172.21.15.28:0/3710621780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6e1350 msgr2=0x55636da9ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6e1350 0x55636da9ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.567+0000 7f806e2e8640 1 -- 172.21.15.28:0/3710621780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636daa0520 con 0x55636d6b8490 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f806e2e8640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8064009580 tx=0x7f8064004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4744186ff231b013 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806400f8b0 con 0x55636d6b8490 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f806000fe50 con 0x55636d6b8490 2024-09-21T05:56:08.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f806400fa50 con 0x55636d6b8490 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f806400fc50 con 0x55636d6b8490 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8076ac0740 10 client.27166 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55636daa2e90 con 0x55636d6b8490 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f806000fe50 con 0x55636d6b8490 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.568+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f8064027070 con 0x55636d6b8490 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 10 client.27166 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 10 client.27166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 20 client.27166 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d69fcd0 0x55636daa6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 4 client.27166 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55636da00db0 con 0x55636d69fcd0 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f806dae7640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d69fcd0 0x55636daa6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f806dae7640 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d69fcd0 0x55636daa6470 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c7a7b5168bd727 server_cookie=ed35c9f597e8c6eb in_seq=0 out_seq=0 2024-09-21T05:56:08.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.569+0000 7f8051ffb640 10 client.27166 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8051ffb640 1 -- 172.21.15.28:0/3710621780 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55636da00db0 con 0x55636d69fcd0 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8051ffb640 10 client.27166 handle_command_reply: tid=0 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 client.27166 shutdown 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 msgr2=0x55636da94050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8064009580 tx=0x7f8064004730 comp rx=0 tx=0).stop 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 shutdown_connections 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d69fcd0 0x55636daa6470 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6e1350 0x55636da9ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6c0670 0x55636da94590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8051ffb640 0 client.27166 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3710621780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da94050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 >> 172.21.15.28:0/3710621780 conn(0x55636d6d5740 msgr2=0x55636da8b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 shutdown_connections 2024-09-21T05:56:08.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 1 -- 172.21.15.28:0/3710621780 wait complete. 2024-09-21T05:56:08.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.570+0000 7f8076ac0740 20 client.27166 trim_cache size 0 max 16384 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 Processor -- start 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 -- start start 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 0x55636da89b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da8da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80681af8f0 con 0x55636d6c0670 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80681b14f0 con 0x55636d6b8490 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f8076ac0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f80681afff0 con 0x55636d6e1350 2024-09-21T05:56:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 0x55636da89b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39370/0 (socket says 172.21.15.28:39370) 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806dae7640 1 -- 172.21.15.28:0/3288060634 learned_addr learned my addr 172.21.15.28:0/3288060634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806eae9640 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da8da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806dae7640 1 -- 172.21.15.28:0/3288060634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 msgr2=0x55636da8da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.588+0000 7f806dae7640 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da8da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f806dae7640 1 -- 172.21.15.28:0/3288060634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 msgr2=0x55636da89b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f806dae7640 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 0x55636da89b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f806dae7640 1 -- 172.21.15.28:0/3288060634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636da86860 con 0x55636d6c0670 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f806e2e8640 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 0x55636da89b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f806dae7640 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8058105380 tx=0x7f805803b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ed57f61ea46e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f8053fff640 1 -- 172.21.15.28:0/3288060634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80580f10b0 con 0x55636d6c0670 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f8053fff640 1 -- 172.21.15.28:0/3288060634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80580ce120 con 0x55636d6c0670 2024-09-21T05:56:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.589+0000 7f8053fff640 1 -- 172.21.15.28:0/3288060634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80580d2100 con 0x55636d6c0670 2024-09-21T05:56:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- 172.21.15.28:0/3288060634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 msgr2=0x55636da8d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8058105380 tx=0x7f805803b460 comp rx=0 tx=0).stop 2024-09-21T05:56:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- 172.21.15.28:0/3288060634 shutdown_connections 2024-09-21T05:56:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6e1350 0x55636da8da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da8d4c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 --2- 172.21.15.28:0/3288060634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6b8490 0x55636da89b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- 172.21.15.28:0/3288060634 >> 172.21.15.28:0/3288060634 conn(0x55636d6d5740 msgr2=0x55636da89460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- 172.21.15.28:0/3288060634 shutdown_connections 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- 172.21.15.28:0/3288060634 wait complete. 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 -- start start 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.590+0000 7f8076ac0740 1 Processor -- start 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da9c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 0x55636da98f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80681b0df0 con 0x55636d6c0670 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80681b06f0 con 0x55636d6e1350 2024-09-21T05:56:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f80681ad8f0 con 0x55636d6b8490 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8076ac0740 10 client.0 _ll_register_callbacks cb 0x7f8064009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806e2e8640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da9c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39374/0 (socket says 172.21.15.28:39374) 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 -- 172.21.15.28:0/465452683 learned_addr learned my addr 172.21.15.28:0/465452683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806eae9640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 0x55636da98f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 -- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 msgr2=0x55636da9c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da9c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 -- 172.21.15.28:0/465452683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 msgr2=0x55636da98f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 0x55636da98f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 -- 172.21.15.28:0/465452683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636da995b0 con 0x55636d6c0670 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806eae9640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 0x55636da98f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806e2e8640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da9c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:08.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f806dae7640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f80581053d0 tx=0x7f80581057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14a5f1b2e2e0808 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.591+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80580f10b0 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f80580e3bd0 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80580ce120 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f80580d2300 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 10 client.17706 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55636db11cc0 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80580ce120 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f80580c9400 con 0x55636d6c0670 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 10 client.17706 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 20 client.17706 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d675080 0x55636db14470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 4 client.17706 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-21T05:56:08.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55636d9fc860 con 0x55636d675080 2024-09-21T05:56:08.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.592+0000 7f806e2e8640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d675080 0x55636db14470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:08.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.593+0000 7f8051ffb640 10 client.17706 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:08.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.593+0000 7f806e2e8640 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d675080 0x55636db14470 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c612d9af66dad53f server_cookie=ebf69472634144c6 in_seq=0 out_seq=0 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8051ffb640 1 -- 172.21.15.28:0/465452683 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55636d9fc860 con 0x55636d675080 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8051ffb640 10 client.17706 handle_command_reply: tid=0 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 client.17706 shutdown 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 msgr2=0x55636da98a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f80581053d0 tx=0x7f80581057d0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 shutdown_connections 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55636d675080 0x55636db14470 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d6e1350 0x55636da98f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55636d6c0670 0x55636da98a00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 --2- 172.21.15.28:0/465452683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55636d6b8490 0x55636da9c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 >> 172.21.15.28:0/465452683 conn(0x55636d6d5740 msgr2=0x55636da89e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.594+0000 7f8051ffb640 0 client.17706 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 shutdown_connections 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f8076ac0740 1 -- 172.21.15.28:0/465452683 wait complete. 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f8076ac0740 20 client.17706 trim_cache size 0 max 16384 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e6db6e23-3bed-41eb-9086-a0b9021d388c", 2024-09-21T05:56:08.601 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-21T05:56:08.602 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 msgr2=0x7f80540b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 0x7f80540b8c10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55636da870b0 tx=0x7f8064005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 msgr2=0x7f806819aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058031a20 comp rx=0 tx=0).stop 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 shutdown_connections 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806810de10 0x7f806819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f80540b6700 0x7f80540b8c10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068105300 0x7f806819aae0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 --2- 172.21.15.28:0/2865783275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80681049b0 0x7f806819a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 >> 172.21.15.28:0/2865783275 conn(0x7f8068079730 msgr2=0x7f80680ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 shutdown_connections 2024-09-21T05:56:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:08.595+0000 7f807054c640 1 -- 172.21.15.28:0/2865783275 wait complete. 2024-09-21T05:56:08.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:08 smithi181 ceph-mon[34819]: pgmap v603: 97 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 82 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:08 smithi129 ceph-mon[34885]: pgmap v603: 97 pgs: 4 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 82 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:09.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:56:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:09 smithi028 ceph-mon[31227]: scrub queued for path: / 2024-09-21T05:56:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:09 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:09 smithi028 ceph-mon[31227]: scrub summary: active paths [/] 2024-09-21T05:56:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:09 smithi028 ceph-mon[31227]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:09.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:09 smithi181 ceph-mon[34819]: scrub queued for path: / 2024-09-21T05:56:09.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:09 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:09.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:09 smithi181 ceph-mon[34819]: scrub summary: active paths [/] 2024-09-21T05:56:09.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:09 smithi181 ceph-mon[34819]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:09.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:09 smithi129 ceph-mon[34885]: scrub queued for path: / 2024-09-21T05:56:09.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:09 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:09.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:09 smithi129 ceph-mon[34885]: scrub summary: active paths [/] 2024-09-21T05:56:09.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:09 smithi129 ceph-mon[34885]: scrub summary: idle+waiting paths [/] 2024-09-21T05:56:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- 172.21.15.28:0/4019929938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210109e00 msgr2=0x7f4210110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 --2- 172.21.15.28:0/4019929938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210109e00 0x7f4210110630 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4208009ca0 tx=0x7f420802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- 172.21.15.28:0/4019929938 shutdown_connections 2024-09-21T05:56:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 --2- 172.21.15.28:0/4019929938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210109e00 0x7f4210110630 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 --2- 172.21.15.28:0/4019929938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42101094b0 0x7f42101098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 --2- 172.21.15.28:0/4019929938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4210103bc0 0x7f4210103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- 172.21.15.28:0/4019929938 >> 172.21.15.28:0/4019929938 conn(0x7f421007b910 msgr2=0x7f421007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- 172.21.15.28:0/4019929938 shutdown_connections 2024-09-21T05:56:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- 172.21.15.28:0/4019929938 wait complete. 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 Processor -- start 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.125+0000 7f4216724640 1 -- start start 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42101094b0 0x7f4210075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4210109e00 0x7f4210076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4210113ac0 con 0x7f4210103bc0 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4210113940 con 0x7f4210109e00 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4216724640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4210113c40 con 0x7f42101094b0 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39400/0 (socket says 172.21.15.28:39400) 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 -- 172.21.15.28:0/2107816740 learned_addr learned my addr 172.21.15.28:0/2107816740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4207fff640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42101094b0 0x7f4210075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4210109e00 0x7f4210076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 -- 172.21.15.28:0/2107816740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42101094b0 msgr2=0x7f4210075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42101094b0 0x7f4210075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 -- 172.21.15.28:0/2107816740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4210109e00 msgr2=0x7f4210076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4210109e00 0x7f4210076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 -- 172.21.15.28:0/2107816740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42101a86f0 con 0x7f4210103bc0 2024-09-21T05:56:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420ffff640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f420000d950 tx=0x7f420000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.126+0000 7f420dffb640 1 -- 172.21.15.28:0/2107816740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4200017070 con 0x7f4210103bc0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.127+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42101a89e0 con 0x7f4210103bc0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.127+0000 7f420dffb640 1 -- 172.21.15.28:0/2107816740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4200009c30 con 0x7f4210103bc0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.127+0000 7f420dffb640 1 -- 172.21.15.28:0/2107816740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420000f620 con 0x7f4210103bc0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.127+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42101a8f70 con 0x7f4210103bc0 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 Processor -- start 2024-09-21T05:56:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 -- start start 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c163cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55870c0d9e50 con 0x55870bd931b0 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55870c0c9280 con 0x55870bd9b390 2024-09-21T05:56:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f421cc98740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55870c0cd090 con 0x55870bdbc070 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f420dffb640 1 -- 172.21.15.28:0/2107816740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f4200010460 con 0x7f4210103bc0 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f4214cc1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f4214cc1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:36876/0 (socket says 172.21.15.28:36876) 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f4214cc1640 1 -- 172.21.15.28:0/4227195700 learned_addr learned my addr 172.21.15.28:0/4227195700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f4207fff640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c163cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.130+0000 7f420ffff640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 -- 172.21.15.28:0/4227195700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 msgr2=0x55870c163270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 -- 172.21.15.28:0/4227195700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 msgr2=0x55870c1637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 -- 172.21.15.28:0/4227195700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870be27650 con 0x55870bd9b390 2024-09-21T05:56:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4214cc1640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f4207fff640 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c163cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f421006b8a0 tx=0x7f41f80113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76dbb820caf791a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f42067fc640 1 -- 172.21.15.28:0/4227195700 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8011de0 con 0x55870bd9b390 2024-09-21T05:56:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.131+0000 7f42067fc640 1 -- 172.21.15.28:0/4227195700 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f8012900 con 0x55870bd9b390 2024-09-21T05:56:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.132+0000 7f420dffb640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 0x7f41fc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.132+0000 7f420dffb640 1 -- 172.21.15.28:0/2107816740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f42000e7470 con 0x7f4210103bc0 2024-09-21T05:56:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.132+0000 7f42067fc640 1 -- 172.21.15.28:0/4227195700 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800f960 con 0x55870bd9b390 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.132+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 0x7f41fc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- 172.21.15.28:0/4227195700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 msgr2=0x55870c163cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c163cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f421006b8a0 tx=0x7f41f80113e0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- 172.21.15.28:0/4227195700 shutdown_connections 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c163cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 --2- 172.21.15.28:0/4227195700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c163270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- 172.21.15.28:0/4227195700 >> 172.21.15.28:0/4227195700 conn(0x55870bdb0460 msgr2=0x55870c161160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 0x7f41fc0b8b90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4208005e30 tx=0x7f4208005c30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- 172.21.15.28:0/4227195700 shutdown_connections 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- 172.21.15.28:0/4227195700 wait complete. 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.133+0000 7f421cc98740 1 -- start start 2024-09-21T05:56:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 Processor -- start 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c16e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c16ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bdbc070 0x55870c16f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55870bf2a700 con 0x55870bd931b0 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55870c064110 con 0x55870bdbc070 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55870c171af0 con 0x55870bd9b390 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c16e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f421cc98740 10 client.0 _ll_register_callbacks cb 0x55870c07d990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f4207fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c16ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c16e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39410/0 (socket says 172.21.15.28:39410) 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f420ffff640 1 -- 172.21.15.28:0/3319750279 learned_addr learned my addr 172.21.15.28:0/3319750279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.134+0000 7f4214cc1640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bdbc070 0x55870c16f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 -- 172.21.15.28:0/3319750279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bdbc070 msgr2=0x55870c16f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bdbc070 0x55870c16f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 -- 172.21.15.28:0/3319750279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 msgr2=0x55870c16e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c16e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 -- 172.21.15.28:0/3319750279 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870c091c40 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f4207fff640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c16ed90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41f800f480 tx=0x7f41f8012680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69454a93e332ff02 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f41ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8028020 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f41f8005050 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.135+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f8012900 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f41f8033050 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 10 client.27184 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55870c17d720 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f8012900 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f41f8004b60 con 0x55870bd9b390 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 10 client.27184 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 10 client.27184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 20 client.27184 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd7a9f0 0x55870c180ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 4 client.27184 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55870c0dbda0 con 0x55870bd7a9f0 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.136+0000 7f420ffff640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd7a9f0 0x55870c180ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.137+0000 7f41ef7fe640 10 client.27184 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.137+0000 7f420ffff640 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd7a9f0 0x55870c180ae0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d801a7b94b85c0 server_cookie=a007d27ac2fa3840 in_seq=0 out_seq=0 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.137+0000 7f41ef7fe640 1 -- 172.21.15.28:0/3319750279 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55870c0dbda0 con 0x55870bd7a9f0 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.137+0000 7f41ef7fe640 10 client.27184 handle_command_reply: tid=0 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 client.27184 shutdown 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 msgr2=0x55870c16ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c16ed90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f41f800f480 tx=0x7f41f8012680 comp rx=0 tx=0).stop 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 shutdown_connections 2024-09-21T05:56:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd7a9f0 0x55870c180ae0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bdbc070 0x55870c16f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c16ed90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f41ef7fe640 0 client.27184 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 --2- 172.21.15.28:0/3319750279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c16e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 >> 172.21.15.28:0/3319750279 conn(0x55870bdb0460 msgr2=0x55870c166400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 shutdown_connections 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 1 -- 172.21.15.28:0/3319750279 wait complete. 2024-09-21T05:56:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.138+0000 7f421cc98740 20 client.27184 trim_cache size 0 max 16384 2024-09-21T05:56:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.155+0000 7f421cc98740 1 Processor -- start 2024-09-21T05:56:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 -- start start 2024-09-21T05:56:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd931b0 0x55870c16fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c160710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42101b1a80 con 0x55870bdbc070 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42101b3680 con 0x55870bd931b0 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f421cc98740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42101b2180 con 0x55870bd9b390 2024-09-21T05:56:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39414/0 (socket says 172.21.15.28:39414) 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 -- 172.21.15.28:0/3515891408 learned_addr learned my addr 172.21.15.28:0/3515891408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4207fff640 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f420ffff640 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd931b0 0x55870c16fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 -- 172.21.15.28:0/3515891408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 msgr2=0x55870c160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c160710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 -- 172.21.15.28:0/3515891408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd931b0 msgr2=0x55870c16fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd931b0 0x55870c16fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 -- 172.21.15.28:0/3515891408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870c170c20 con 0x55870bdbc070 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.156+0000 7f4214cc1640 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4208032ea0 tx=0x7f4208044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef34c4caa5f84989 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.157+0000 7f4205ffb640 1 -- 172.21.15.28:0/3515891408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208003970 con 0x55870bdbc070 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.157+0000 7f4205ffb640 1 -- 172.21.15.28:0/3515891408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4208036040 con 0x55870bdbc070 2024-09-21T05:56:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.157+0000 7f4205ffb640 1 -- 172.21.15.28:0/3515891408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420802f660 con 0x55870bdbc070 2024-09-21T05:56:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.157+0000 7f421cc98740 1 -- 172.21.15.28:0/3515891408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 msgr2=0x55870c160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4208032ea0 tx=0x7f4208044000 comp rx=0 tx=0).stop 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 -- 172.21.15.28:0/3515891408 shutdown_connections 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bdbc070 0x55870c160c50 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bd9b390 0x55870c160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 --2- 172.21.15.28:0/3515891408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd931b0 0x55870c16fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 -- 172.21.15.28:0/3515891408 >> 172.21.15.28:0/3515891408 conn(0x55870bdb0460 msgr2=0x55870c164c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 -- 172.21.15.28:0/3515891408 shutdown_connections 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 -- 172.21.15.28:0/3515891408 wait complete. 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 -- start start 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.158+0000 7f421cc98740 1 Processor -- start 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c181900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c181f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42101b2f80 con 0x55870bd931b0 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42101b2880 con 0x55870bd9b390 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42101afa80 con 0x55870bdbc070 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f421cc98740 10 client.0 _ll_register_callbacks cb 0x7f42000eae90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c181f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f420ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39428/0 (socket says 172.21.15.28:39428) 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f420ffff640 1 -- 172.21.15.28:0/2477486481 learned_addr learned my addr 172.21.15.28:0/2477486481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4207fff640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c181900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 -- 172.21.15.28:0/2477486481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 msgr2=0x55870c181900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c181900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 -- 172.21.15.28:0/2477486481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 msgr2=0x55870c1843e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 -- 172.21.15.28:0/2477486481 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870c1826c0 con 0x55870bdbc070 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f420ffff640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.159+0000 7f4214cc1640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c181f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2133cddb1b7a3f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420805b070 con 0x55870bdbc070 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4208036040 con 0x55870bdbc070 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f420802f860 con 0x55870bdbc070 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4208067050 con 0x55870bdbc070 2024-09-21T05:56:10.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 10 client.27190 fetch_fsmap learned FSMap version 49 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55870c1eb2b0 con 0x55870bdbc070 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4208036040 con 0x55870bdbc070 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 49) ==== 8382+0+0 (secure 0 0 0) 0x7f4208030c40 con 0x55870bdbc070 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 10 client.27190 fetch_fsmap finished waiting for FSMap version 49 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 10 client.27190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 20 client.27190 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd4fda0 0x55870c1eda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 4 client.27190 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.160+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55870c0d7850 con 0x55870bd4fda0 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f420ffff640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd4fda0 0x55870c1eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f41ef7fe640 10 client.27190 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:10.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f420ffff640 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd4fda0 0x55870c1eda50 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d7d56d5a9ad08f3 server_cookie=89676c5ea45c074d in_seq=0 out_seq=0 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f41ef7fe640 1 -- 172.21.15.28:0/2477486481 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4262 (crc 0 0 0) 0x55870c0d7850 con 0x55870bd4fda0 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f41ef7fe640 10 client.27190 handle_command_reply: tid=0 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.161+0000 7f421cc98740 1 client.27190 shutdown 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 msgr2=0x55870c181f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c181f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208004960 comp rx=0 tx=0).stop 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 shutdown_connections 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x55870bd4fda0 0x55870c1eda50 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55870bdbc070 0x55870c181f50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55870bd9b390 0x55870c181900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f41ef7fe640 0 client.27190 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 --2- 172.21.15.28:0/2477486481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55870bd931b0 0x55870c1843e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 >> 172.21.15.28:0/2477486481 conn(0x55870bdb0460 msgr2=0x55870c164c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 shutdown_connections 2024-09-21T05:56:10.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 1 -- 172.21.15.28:0/2477486481 wait complete. 2024-09-21T05:56:10.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f421cc98740 20 client.27190 trim_cache size 0 max 16384 2024-09-21T05:56:10.171 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:10.171 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "e6db6e23-3bed-41eb-9086-a0b9021d388c": { 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e6db6e23-3bed-41eb-9086-a0b9021d388c", 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "e6db6e23-3bed-41eb-9086-a0b9021d388c": { 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:56:08.593733+0000", 2024-09-21T05:56:10.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.174 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.175 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.176 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "dcf8e535-cb94-40fa-83e5-e2ec6e2e97ed": { 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:54.843004+0000", 2024-09-21T05:56:10.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.180 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:09.151907+0000", 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:10.181 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 msgr2=0x7f41fc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 0x7f41fc0b8b90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4208005e30 tx=0x7f4208005c30 comp rx=0 tx=0).stop 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 msgr2=0x7f4210075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.162+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f420000d950 tx=0x7f420000de10 comp rx=0 tx=0).stop 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 shutdown_connections 2024-09-21T05:56:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4210109e00 0x7f4210076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42101094b0 0x7f4210075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41fc0b6680 0x7f41fc0b8b90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 --2- 172.21.15.28:0/2107816740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4210103bc0 0x7f4210075710 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 >> 172.21.15.28:0/2107816740 conn(0x7f421007b910 msgr2=0x7f42101074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 shutdown_connections 2024-09-21T05:56:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:10.163+0000 7f4216724640 1 -- 172.21.15.28:0/2107816740 wait complete. 2024-09-21T05:56:10.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:10 smithi028 ceph-mon[31227]: pgmap v604: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:10.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:10 smithi181 ceph-mon[34819]: pgmap v604: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:10.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:10 smithi129 ceph-mon[34885]: pgmap v604: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:11.004 INFO:tasks.cephfs.filesystem:scrub status for tag:e6db6e23-3bed-41eb-9086-a0b9021d388c - {'path': '/', 'tag': 'e6db6e23-3bed-41eb-9086-a0b9021d388c', 'options': 'recursive,force'} 2024-09-21T05:56:11.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:11 smithi181 ceph-mon[34819]: scrub summary: idle 2024-09-21T05:56:11.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:11 smithi028 ceph-mon[31227]: scrub summary: idle 2024-09-21T05:56:11.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:11 smithi129 ceph-mon[34885]: scrub summary: idle 2024-09-21T05:56:12.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:12 smithi181 ceph-mon[34819]: pgmap v605: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:12 smithi028 ceph-mon[31227]: pgmap v605: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:12 smithi129 ceph-mon[34885]: pgmap v605: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 18 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T05:56:13.811 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:56:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x[31456]: 2024-09-21T05:56:13.457+0000 7f3faf3a0640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-21T05:56:14.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:14 smithi181 ceph-mon[34819]: pgmap v606: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:56:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:14 smithi028 ceph-mon[31227]: pgmap v606: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:56:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:14 smithi129 ceph-mon[34885]: pgmap v606: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-09-21T05:56:16.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:16 smithi181 ceph-mon[34819]: pgmap v607: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:56:16.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:16 smithi028 ceph-mon[31227]: pgmap v607: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:56:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:16 smithi129 ceph-mon[34885]: pgmap v607: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-21T05:56:18.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:18 smithi181 ceph-mon[34819]: pgmap v608: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 10 op/s 2024-09-21T05:56:18.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:18 smithi028 ceph-mon[31227]: pgmap v608: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 10 op/s 2024-09-21T05:56:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:18 smithi129 ceph-mon[34885]: pgmap v608: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 193 B/s wr, 10 op/s 2024-09-21T05:56:20.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:20 smithi181 ceph-mon[34819]: pgmap v609: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:56:20.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:20 smithi028 ceph-mon[31227]: pgmap v609: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:56:20.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:20 smithi129 ceph-mon[34885]: pgmap v609: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-09-21T05:56:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:22 smithi181 ceph-mon[34819]: pgmap v610: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-21T05:56:22.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:22 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:22 smithi028 ceph-mon[31227]: pgmap v610: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-21T05:56:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:22 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:22 smithi129 ceph-mon[34885]: pgmap v610: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-21T05:56:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:22 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:24.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:24 smithi181 ceph-mon[34819]: pgmap v611: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T05:56:24.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:24 smithi028 ceph-mon[31227]: pgmap v611: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T05:56:24.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:24 smithi129 ceph-mon[34885]: pgmap v611: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T05:56:25.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:25 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:25.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:25 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:25.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:25 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:26.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:26 smithi129 ceph-mon[34885]: pgmap v612: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:26.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:26 smithi181 ceph-mon[34819]: pgmap v612: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:26 smithi028 ceph-mon[31227]: pgmap v612: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:28.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:28 smithi181 ceph-mon[34819]: pgmap v613: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:28.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:28 smithi028 ceph-mon[31227]: pgmap v613: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:28 smithi129 ceph-mon[34885]: pgmap v613: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:30.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:30 smithi181 ceph-mon[34819]: pgmap v614: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:30.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:30 smithi028 ceph-mon[31227]: pgmap v614: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:30.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:30 smithi129 ceph-mon[34885]: pgmap v614: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:32.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:32 smithi181 ceph-mon[34819]: pgmap v615: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:32.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:32 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:32 smithi028 ceph-mon[31227]: pgmap v615: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:32 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:32 smithi129 ceph-mon[34885]: pgmap v615: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:32.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:32 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:34.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:34 smithi181 ceph-mon[34819]: pgmap v616: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:56:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:34 smithi028 ceph-mon[31227]: pgmap v616: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:56:34.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:34 smithi129 ceph-mon[34885]: pgmap v616: 97 pgs: 23 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 34 active+clean; 17 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T05:56:35.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:35 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:35.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:35 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:35.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:35 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:36.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:36 smithi181 ceph-mon[34819]: pgmap v617: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 17 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:36.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:36 smithi028 ceph-mon[31227]: pgmap v617: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 17 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:36.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:36 smithi129 ceph-mon[34885]: pgmap v617: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 17 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-21T05:56:38.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:38 smithi181 ceph-mon[34819]: pgmap v618: 97 pgs: 19 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 42 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:38.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:38 smithi028 ceph-mon[31227]: pgmap v618: 97 pgs: 19 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 42 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:38.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:38 smithi129 ceph-mon[34885]: pgmap v618: 97 pgs: 19 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 42 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:39.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T05:56:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 -- 172.21.15.28:0/1934849263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104930 msgr2=0x7f7fc0104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1934849263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104930 0x7f7fc0104d40 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009900 tx=0x7f7fb002a410 comp rx=0 tx=0).stop 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 -- 172.21.15.28:0/1934849263 shutdown_connections 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1934849263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 0x7f7fc0110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1934849263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0105280 0x7f7fc010d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1934849263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0104930 0x7f7fc0104d40 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 -- 172.21.15.28:0/1934849263 >> 172.21.15.28:0/1934849263 conn(0x7f7fc00fe3c0 msgr2=0x7f7fc0100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 -- 172.21.15.28:0/1934849263 shutdown_connections 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.544+0000 7f7fc612b640 1 -- 172.21.15.28:0/1934849263 wait complete. 2024-09-21T05:56:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 Processor -- start 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 -- start start 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0104930 0x7f7fc019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 0x7f7fc019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc01139f0 con 0x7f7fc0105280 2024-09-21T05:56:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0113870 con 0x7f7fc0104930 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fc612b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0113b70 con 0x7f7fc010dd90 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49664/0 (socket says 172.21.15.28:49664) 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fbeffd640 1 -- 172.21.15.28:0/1160221361 learned_addr learned my addr 172.21.15.28:0/1160221361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fbf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0104930 0x7f7fc019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.545+0000 7f7fbffff640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 0x7f7fc019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 -- 172.21.15.28:0/1160221361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 msgr2=0x7f7fc019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 0x7f7fc019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 -- 172.21.15.28:0/1160221361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0104930 msgr2=0x7f7fc019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0104930 0x7f7fc019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 -- 172.21.15.28:0/1160221361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01a0340 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbeffd640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400e9f0 tx=0x7f7fb400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400ccc0 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb40108e0 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01a0630 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4018860 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.546+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01a6dc0 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.547+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc0104d40 con 0x7f7fc0105280 2024-09-21T05:56:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.549+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7fb4010430 con 0x7f7fc0105280 2024-09-21T05:56:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.550+0000 7f7fbcff9640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 0x7f7fa80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.550+0000 7f7fbf7fe640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 0x7f7fa80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.550+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f7fb40e7eb0 con 0x7f7fc0105280 2024-09-21T05:56:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.551+0000 7f7fbf7fe640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 0x7f7fa80b8b90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0031440 tx=0x7f7fb0009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.553+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7fb40b27b0 con 0x7f7fc0105280 2024-09-21T05:56:39.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.783+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7fc0100620 con 0x7f7fa80b6680 2024-09-21T05:56:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.793+0000 7f7fbcff9640 1 -- 172.21.15.28:0/1160221361 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+349 (secure 0 0 0) 0x7f7fc0100620 con 0x7f7fa80b6680 2024-09-21T05:56:39.795 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T05:46:38", "first": "2024-09-21T05:47:00", "last": "2024-09-21T05:56:00", "last_pruned": "2024-09-21T05:56:00", "created_count": 9, "pruned_count": 3, "active": true} 2024-09-21T05:56:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 msgr2=0x7f7fa80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 0x7f7fa80b8b90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0031440 tx=0x7f7fb0009560 comp rx=0 tx=0).stop 2024-09-21T05:56:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 msgr2=0x7f7fc019a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400e9f0 tx=0x7f7fb400eeb0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 shutdown_connections 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7fc010dd90 0x7f7fc019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f7fa80b6680 0x7f7fa80b8b90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc0105280 0x7f7fc019a9b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 --2- 172.21.15.28:0/1160221361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fc0104930 0x7f7fc019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 >> 172.21.15.28:0/1160221361 conn(0x7f7fc00fe3c0 msgr2=0x7f7fc00fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 shutdown_connections 2024-09-21T05:56:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.795+0000 7f7fc612b640 1 -- 172.21.15.28:0/1160221361 wait complete. 2024-09-21T05:56:39.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 -- 172.21.15.28:0/3492869107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 msgr2=0x7f3dcc102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/3492869107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc102dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009900 tx=0x7f3dbc02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 -- 172.21.15.28:0/3492869107 shutdown_connections 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/3492869107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc108c30 0x7f3dcc10f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/3492869107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/3492869107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc102dc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 -- 172.21.15.28:0/3492869107 >> 172.21.15.28:0/3492869107 conn(0x7f3dcc07a740 msgr2=0x7f3dcc07ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 -- 172.21.15.28:0/3492869107 shutdown_connections 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.927+0000 7f3dd1cde640 1 -- 172.21.15.28:0/3492869107 wait complete. 2024-09-21T05:56:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.928+0000 7f3dd1cde640 1 Processor -- start 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.928+0000 7f3dd1cde640 1 -- start start 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc108c30 0x7f3dcc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc1128f0 con 0x7f3dcc1029b0 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc112770 con 0x7f3dcc1082e0 2024-09-21T05:56:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dd1cde640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc112a70 con 0x7f3dcc108c30 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49684/0 (socket says 172.21.15.28:49684) 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 -- 172.21.15.28:0/1434850749 learned_addr learned my addr 172.21.15.28:0/1434850749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcbfff640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc108c30 0x7f3dcc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcaffd640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 -- 172.21.15.28:0/1434850749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc108c30 msgr2=0x7f3dcc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc108c30 0x7f3dcc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 -- 172.21.15.28:0/1434850749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc1082e0 msgr2=0x7f3dcc199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 -- 172.21.15.28:0/1434850749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc19f2a0 con 0x7f3dcc1029b0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dcb7fe640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005940 tx=0x7f3dbc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.929+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc03f030 con 0x7f3dcc1029b0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.930+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dbc003cb0 con 0x7f3dcc1029b0 2024-09-21T05:56:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.930+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc035070 con 0x7f3dcc1029b0 2024-09-21T05:56:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.930+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc19f530 con 0x7f3dcc1029b0 2024-09-21T05:56:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.930+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc1a5d20 con 0x7f3dcc1029b0 2024-09-21T05:56:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.931+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc102dc0 con 0x7f3dcc1029b0 2024-09-21T05:56:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.932+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f3dbc042030 con 0x7f3dcc1029b0 2024-09-21T05:56:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.933+0000 7f3dc8ff9640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 0x7f3db40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.933+0000 7f3dcaffd640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 0x7f3db40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.933+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f3dbc108230 con 0x7f3dcc1029b0 2024-09-21T05:56:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.934+0000 7f3dcaffd640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 0x7f3db40b8b90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc07b1c0 tx=0x7f3dc000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:39.937+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3dbc0d2b30 con 0x7f3dcc1029b0 2024-09-21T05:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.159+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 --> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f3dcc10aa30 con 0x7f3db40b6680 2024-09-21T05:56:40.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.198+0000 7f3dc8ff9640 1 -- 172.21.15.28:0/1434850749 <== mgr.14223 v2:172.21.15.28:6800/4211147350 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f3dcc10aa30 con 0x7f3db40b6680 2024-09-21T05:56:40.200 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-09-21T05:56:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 msgr2=0x7f3db40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 0x7f3db40b8b90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc07b1c0 tx=0x7f3dc000b390 comp rx=0 tx=0).stop 2024-09-21T05:56:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 msgr2=0x7f3dcc199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005940 tx=0x7f3dbc0042f0 comp rx=0 tx=0).stop 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 shutdown_connections 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dcc108c30 0x7f3dcc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f3db40b6680 0x7f3db40b8b90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 --2- 172.21.15.28:0/1434850749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dcc1029b0 0x7f3dcc199370 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 >> 172.21.15.28:0/1434850749 conn(0x7f3dcc07a740 msgr2=0x7f3dcc1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 shutdown_connections 2024-09-21T05:56:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:40.200+0000 7f3dd1cde640 1 -- 172.21.15.28:0/1434850749 wait complete. 2024-09-21T05:56:40.214 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-21T05:56:40.228 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-21T05:56:40.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:40 smithi181 ceph-mon[34819]: pgmap v619: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:40.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:40 smithi181 ceph-mon[34819]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:40.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:40 smithi028 ceph-mon[31227]: pgmap v619: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:40.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:40 smithi028 ceph-mon[31227]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:40 smithi129 ceph-mon[34885]: pgmap v619: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:56:40.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:40 smithi129 ceph-mon[34885]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T05:56:41.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 scrub status 2024-09-21T05:56:41.272 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:41 smithi028 ceph-mon[31227]: from='client.17730 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:41 smithi028 ceph-mon[31227]: from='client.17736 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:41.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:41 smithi181 ceph-mon[34819]: from='client.17730 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:41.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:41 smithi181 ceph-mon[34819]: from='client.17736 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.804+0000 7f972ba4e640 1 -- 172.21.15.28:0/3620451832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724110180 msgr2=0x7f97241125a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.804+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3620451832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724110180 0x7f97241125a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f971c009ca0 tx=0x7f971c02a950 comp rx=0 tx=0).stop 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.804+0000 7f972ba4e640 1 -- 172.21.15.28:0/3620451832 shutdown_connections 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.804+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3620451832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724110180 0x7f97241125a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3620451832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724102f20 0x7f972410fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3620451832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 0x7f97241029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 -- 172.21.15.28:0/3620451832 >> 172.21.15.28:0/3620451832 conn(0x7f9724079730 msgr2=0x7f9724079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 -- 172.21.15.28:0/3620451832 shutdown_connections 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 -- 172.21.15.28:0/3620451832 wait complete. 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 Processor -- start 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.805+0000 7f972ba4e640 1 -- start start 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 0x7f972419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724110180 0x7f97241a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9724115c70 con 0x7f9724102f20 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9724115af0 con 0x7f9724110180 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f972ba4e640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9724115df0 con 0x7f97241025d0 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49710/0 (socket says 172.21.15.28:49710) 2024-09-21T05:56:41.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 -- 172.21.15.28:0/3218536112 learned_addr learned my addr 172.21.15.28:0/3218536112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f97297ea640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 0x7f972419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9729feb640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724110180 0x7f97241a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 -- 172.21.15.28:0/3218536112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 msgr2=0x7f972419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 0x7f972419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 -- 172.21.15.28:0/3218536112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724110180 msgr2=0x7f97241a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724110180 0x7f97241a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 -- 172.21.15.28:0/3218536112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97241a2630 con 0x7f9724102f20 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.806+0000 7f9728fe9640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f971400e9f0 tx=0x7f971400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.807+0000 7f971a7fc640 1 -- 172.21.15.28:0/3218536112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971400cd80 con 0x7f9724102f20 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.807+0000 7f971a7fc640 1 -- 172.21.15.28:0/3218536112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97140108e0 con 0x7f9724102f20 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.807+0000 7f971a7fc640 1 -- 172.21.15.28:0/3218536112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714018980 con 0x7f9724102f20 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.807+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97241a2920 con 0x7f9724102f20 2024-09-21T05:56:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.807+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97241a9080 con 0x7f9724102f20 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 Processor -- start 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 -- start start 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd4b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f983390 0x557d9fd4bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d9fcc1e50 con 0x557d9f983390 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d9fcb1280 con 0x557d9f97b1b0 2024-09-21T05:56:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9731fc2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d9fcb5090 con 0x557d9f9a4070 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9729feb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f97297ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd4b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9729feb640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:56848/0 (socket says 172.21.15.28:56848) 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f9729feb640 1 -- 172.21.15.28:0/188340642 learned_addr learned my addr 172.21.15.28:0/188340642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.810+0000 7f971a7fc640 1 -- 172.21.15.28:0/3218536112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9714010430 con 0x7f9724102f20 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 -- 172.21.15.28:0/188340642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 msgr2=0x557d9fd4b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 -- 172.21.15.28:0/188340642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f983390 msgr2=0x557d9fd4bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f983390 0x557d9fd4bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 -- 172.21.15.28:0/188340642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d9fa0f650 con 0x557d9f97b1b0 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f9729feb640 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.811+0000 7f97297ea640 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd4b7b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9720009c20 tx=0x7f9720011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e1b13bf2f04af5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f970f7fe640 1 -- 172.21.15.28:0/188340642 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720009dd0 con 0x557d9f97b1b0 2024-09-21T05:56:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f970f7fe640 1 -- 172.21.15.28:0/188340642 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9720005020 con 0x557d9f97b1b0 2024-09-21T05:56:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f970f7fe640 1 -- 172.21.15.28:0/188340642 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720004ac0 con 0x557d9f97b1b0 2024-09-21T05:56:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f971a7fc640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 0x7f97100b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f9729feb640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 0x7f97100b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.812+0000 7f971a7fc640 1 -- 172.21.15.28:0/3218536112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f97140e7ab0 con 0x7f9724102f20 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 -- 172.21.15.28:0/188340642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 msgr2=0x557d9fd4b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9729feb640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 0x7f97100b8b90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f971c0094e0 tx=0x7f971c005d80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd4b7b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9720009c20 tx=0x7f9720011810 comp rx=0 tx=0).stop 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 -- 172.21.15.28:0/188340642 shutdown_connections 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f983390 0x557d9fd4bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd4b7b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 --2- 172.21.15.28:0/188340642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd4b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 -- 172.21.15.28:0/188340642 >> 172.21.15.28:0/188340642 conn(0x557d9f998460 msgr2=0x557d9fd49160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 -- 172.21.15.28:0/188340642 shutdown_connections 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.813+0000 7f9731fc2740 1 -- 172.21.15.28:0/188340642 wait complete. 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.814+0000 7f9731fc2740 1 -- start start 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.814+0000 7f9731fc2740 1 Processor -- start 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.814+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd568a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.814+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.814+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9731fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d9fb12700 con 0x557d9f97b1b0 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9731fc2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d9fc4c110 con 0x557d9f9a4070 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9731fc2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d9fd59a50 con 0x557d9f983390 2024-09-21T05:56:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9729feb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f97297ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9729feb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:44004/0 (socket says 172.21.15.28:44004) 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:56850/0 (socket says 172.21.15.28:56850) 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9729feb640 1 -- 172.21.15.28:0/2082240447 learned_addr learned my addr 172.21.15.28:0/2082240447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9731fc2740 10 client.0 _ll_register_callbacks cb 0x557d9fc65990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 -- 172.21.15.28:0/2082240447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 msgr2=0x557d9fd57360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 -- 172.21.15.28:0/2082240447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 msgr2=0x557d9fd568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 -- 172.21.15.28:0/2082240447 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d9fc79c40 con 0x557d9f983390 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9729feb640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f9728fe9640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f97140e51d0 tx=0x7f9714010cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8de1de8b985fd3c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.815+0000 7f970cff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140ad040 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f9714018b20 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9720005020 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f9714010720 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 10 client.27214 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557d9fd65730 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9720005020 con 0x557d9f983390 2024-09-21T05:56:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f97140d3020 con 0x557d9f983390 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 10 client.27214 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 10 client.27214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.816+0000 7f9731fc2740 20 client.27214 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f9629f0 0x557d9fd68d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f9731fc2740 4 client.27214 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d9fcc3da0 con 0x557d9f9629f0 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f97297ea640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f9629f0 0x557d9fd68d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f970cff9640 10 client.27214 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.817+0000 7f97297ea640 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f9629f0 0x557d9fd68d10 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e2fad745f9d361 server_cookie=575d73c293415def in_seq=0 out_seq=0 2024-09-21T05:56:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f970cff9640 1 -- 172.21.15.28:0/2082240447 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d9fcc3da0 con 0x557d9f9629f0 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f970cff9640 10 client.27214 handle_command_reply: tid=0 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 client.27214 shutdown 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 msgr2=0x557d9fd56de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f97140e51d0 tx=0x7f9714010cc0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 shutdown_connections 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f9629f0 0x557d9fd68d10 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f9a4070 0x557d9fd57360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd56de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 --2- 172.21.15.28:0/2082240447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 >> 172.21.15.28:0/2082240447 conn(0x557d9f998460 msgr2=0x557d9fd4e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.818+0000 7f970cff9640 0 client.27214 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.819+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 shutdown_connections 2024-09-21T05:56:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.819+0000 7f9731fc2740 1 -- 172.21.15.28:0/2082240447 wait complete. 2024-09-21T05:56:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.819+0000 7f9731fc2740 20 client.27214 trim_cache size 0 max 16384 2024-09-21T05:56:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.837+0000 7f9731fc2740 1 Processor -- start 2024-09-21T05:56:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.837+0000 7f9731fc2740 1 -- start start 2024-09-21T05:56:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 0x557d9fd584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd48710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ad610 con 0x557d9f97b1b0 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97241af150 con 0x557d9f983390 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9731fc2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97241adce0 con 0x557d9f9a4070 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f97297ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 0x557d9fd584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f97297ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49736/0 (socket says 172.21.15.28:49736) 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f97297ea640 1 -- 172.21.15.28:0/616138378 learned_addr learned my addr 172.21.15.28:0/616138378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9729feb640 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd48710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 -- 172.21.15.28:0/616138378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 msgr2=0x557d9fd48710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd48710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 -- 172.21.15.28:0/616138378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 msgr2=0x557d9fd4cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.838+0000 7f9728fe9640 1 -- 172.21.15.28:0/616138378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d9fd57940 con 0x557d9f983390 2024-09-21T05:56:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.839+0000 7f97297ea640 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.839+0000 7f9728fe9640 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 0x557d9fd584d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97140a7fd0 tx=0x7f9714000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8ff8d48df35bef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.839+0000 7f970effd640 1 -- 172.21.15.28:0/616138378 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140ad3e0 con 0x557d9f983390 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.839+0000 7f970effd640 1 -- 172.21.15.28:0/616138378 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97140adfd0 con 0x557d9f983390 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.839+0000 7f970effd640 1 -- 172.21.15.28:0/616138378 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140a95a0 con 0x557d9f983390 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- 172.21.15.28:0/616138378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 msgr2=0x557d9fd584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 0x557d9fd584d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97140a7fd0 tx=0x7f9714000bc0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- 172.21.15.28:0/616138378 shutdown_connections 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f9a4070 0x557d9fd48710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f983390 0x557d9fd584d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 --2- 172.21.15.28:0/616138378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f97b1b0 0x557d9fd4cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- 172.21.15.28:0/616138378 >> 172.21.15.28:0/616138378 conn(0x557d9f998460 msgr2=0x557d9fd57dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- 172.21.15.28:0/616138378 shutdown_connections 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- 172.21.15.28:0/616138378 wait complete. 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.840+0000 7f9731fc2740 1 -- start start 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 Processor -- start 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd5ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd5b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97241aea80 con 0x557d9f9a4070 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ae3b0 con 0x557d9f97b1b0 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ab630 con 0x557d9f983390 2024-09-21T05:56:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9731fc2740 10 client.0 _ll_register_callbacks cb 0x7f9714009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9729feb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9728fe9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd5b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9729feb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49748/0 (socket says 172.21.15.28:49748) 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f9729feb640 1 -- 172.21.15.28:0/224679275 learned_addr learned my addr 172.21.15.28:0/224679275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.841+0000 7f97297ea640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd5ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 -- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 msgr2=0x557d9fd5b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd5b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 -- 172.21.15.28:0/224679275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 msgr2=0x557d9fd5ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd5ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 -- 172.21.15.28:0/224679275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d9fdd26d0 con 0x557d9f9a4070 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9728fe9640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd5b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9729feb640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f971c006010 tx=0x7f971c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8142cf558d6aff59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f970cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c034040 con 0x557d9f9a4070 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97140adfd0 con 0x557d9f9a4070 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f971c041030 con 0x557d9f9a4070 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f971c004250 con 0x557d9f9a4070 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9731fc2740 10 client.17751 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.842+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557d9fdd38f0 con 0x557d9f9a4070 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97140adfd0 con 0x557d9f9a4070 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f971c030b20 con 0x557d9f9a4070 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 10 client.17751 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 10 client.17751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 20 client.17751 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f937da0 0x557d9fdd6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 4 client.17751 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557d9fcbf850 con 0x557d9f937da0 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.843+0000 7f97297ea640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f937da0 0x557d9fdd6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f970cff9640 10 client.17751 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f97297ea640 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f937da0 0x557d9fdd6110 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608df84b0f19d1f7 server_cookie=b7a1bf94d9cce66 in_seq=0 out_seq=0 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f970cff9640 1 -- 172.21.15.28:0/224679275 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4066 (crc 0 0 0) 0x557d9fcbf850 con 0x557d9f937da0 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f970cff9640 10 client.17751 handle_command_reply: tid=0 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 client.17751 shutdown 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 msgr2=0x557d9fd5b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f971c006010 tx=0x7f971c02a950 comp rx=0 tx=0).stop 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 shutdown_connections 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x557d9f937da0 0x557d9fdd6110 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d9f9a4070 0x557d9fd5b6a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d9f983390 0x557d9fd5b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 --2- 172.21.15.28:0/224679275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d9f97b1b0 0x557d9fd5ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 >> 172.21.15.28:0/224679275 conn(0x557d9f998460 msgr2=0x557d9fd62420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f970cff9640 0 client.17751 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 shutdown_connections 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.844+0000 7f9731fc2740 1 -- 172.21.15.28:0/224679275 wait complete. 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f9731fc2740 20 client.17751 trim_cache size 0 max 16384 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {}, 2024-09-21T05:56:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_stats": { 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "e6db6e23-3bed-41eb-9086-a0b9021d388c": { 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:56:08.593733+0000", 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "87d75cd6-0f75-4c86-962f-fa34714c6cd1": { 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:46:38.930479+0000", 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "778a6c00-a466-4250-b8ce-492ba85cbd6a": { 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:54.561891+0000", 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "72636285-d802-4799-9af4-6cabaeb119f9": { 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:39.649204+0000", 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "ecc4b123-4671-4fe2-b7f5-102d3296a8aa": { 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:48:10.753551+0000", 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "cabb93db-81c7-44a8-a5df-72df2a2e8078": { 2024-09-21T05:56:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:51:50.539942+0000", 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "45ebe5b7-9e03-4b06-b4b1-cee561706b06": { 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:49:00.708503+0000", 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "dcf8e535-cb94-40fa-83e5-e2ec6e2e97ed": { 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:54.843004+0000", 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "bacb5e14-d0ad-4b86-a228-c9c40a2eea63": { 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:50:55.863085+0000", 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "79a01b4c-79fc-4f8e-b68d-3e9d8b7c6941": { 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:47:25.590499+0000", 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "cc0adacb-ac45-4c09-9c6d-8bf3a46edf88": { 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:54:23.548812+0000", 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "c989f3e2-1085-4489-9f7a-2cd347e5ea3a": { 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:52:44.706818+0000", 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "b3760c46-b2b6-415c-99f0-96d4557921a0": { 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:53:36.366760+0000", 2024-09-21T05:56:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "2319b9a4-11f3-496b-9fc8-8eb7f3571f92": { 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "start_time": "2024-09-21T05:55:09.151907+0000", 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_passed": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_failed": 0, 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_skipped": 0 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:41.859 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 msgr2=0x7f97100b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 0x7f97100b8b90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f971c0094e0 tx=0x7f971c005d80 comp rx=0 tx=0).stop 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 msgr2=0x7f972419cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f971400e9f0 tx=0x7f971400eeb0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 shutdown_connections 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9724110180 0x7f97241a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f97100b6680 0x7f97100b8b90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9724102f20 0x7f972419cc60 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 --2- 172.21.15.28:0/3218536112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97241025d0 0x7f972419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 >> 172.21.15.28:0/3218536112 conn(0x7f9724079730 msgr2=0x7f9724109d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 shutdown_connections 2024-09-21T05:56:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:41.845+0000 7f972ba4e640 1 -- 172.21.15.28:0/3218536112 wait complete. 2024-09-21T05:56:41.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:41 smithi129 ceph-mon[34885]: from='client.17730 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:41.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:41 smithi129 ceph-mon[34885]: from='client.17736 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T05:56:42.643 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-21T05:56:42.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:56:42.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:42 smithi181 ceph-mon[34819]: pgmap v620: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:42.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:42 smithi028 ceph-mon[31227]: pgmap v620: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:42.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:42 smithi129 ceph-mon[34885]: pgmap v620: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:56:42.927 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 -- 172.21.15.28:0/1372513975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4108cf0 msgr2=0x7f17d410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 --2- 172.21.15.28:0/1372513975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4108cf0 0x7f17d410f520 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009900 tx=0x7f17cc02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 -- 172.21.15.28:0/1372513975 shutdown_connections 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 --2- 172.21.15.28:0/1372513975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4108cf0 0x7f17d410f520 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 --2- 172.21.15.28:0/1372513975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d41083a0 0x7f17d41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 --2- 172.21.15.28:0/1372513975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d4102a60 0x7f17d4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 -- 172.21.15.28:0/1372513975 >> 172.21.15.28:0/1372513975 conn(0x7f17d4079730 msgr2=0x7f17d4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 -- 172.21.15.28:0/1372513975 shutdown_connections 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.451+0000 7f17dbd83640 1 -- 172.21.15.28:0/1372513975 wait complete. 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 Processor -- start 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 -- start start 2024-09-21T05:56:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 0x7f17d419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d4108cf0 0x7f17d419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4112960 con 0x7f17d4102a60 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17d41127e0 con 0x7f17d4108cf0 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17dbd83640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4112ae0 con 0x7f17d41083a0 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17d9b1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17d9b1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49762/0 (socket says 172.21.15.28:49762) 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17d9b1f640 1 -- 172.21.15.28:0/4124513171 learned_addr learned my addr 172.21.15.28:0/4124513171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.452+0000 7f17da320640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d4108cf0 0x7f17d419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d931e640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 0x7f17d419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 -- 172.21.15.28:0/4124513171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 msgr2=0x7f17d419b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 0x7f17d419b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 -- 172.21.15.28:0/4124513171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d4108cf0 msgr2=0x7f17d419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d4108cf0 0x7f17d419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 -- 172.21.15.28:0/4124513171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d41a6520 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d931e640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 0x7f17d419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17d9b1f640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f17d000eb90 tx=0x7f17d000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004d20 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d000ee90 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d41a6810 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0010670 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.453+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d41a6d00 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.454+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d4102e70 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.456+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f17d0010810 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.456+0000 7f17caffd640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 0x7f17c00b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.457+0000 7f17d931e640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 0x7f17c00b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.457+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f17d00e7fd0 con 0x7f17d4102a60 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.457+0000 7f17d931e640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 0x7f17c00b8bc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f17d4105f80 tx=0x7f17c4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.460+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f17d0056710 con 0x7f17d4102a60 2024-09-21T05:56:43.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.694+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17d406b270 con 0x7f17d4102a60 2024-09-21T05:56:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.695+0000 7f17caffd640 1 -- 172.21.15.28:0/4124513171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11986 (secure 0 0 0) 0x7f17d00ae4f0 con 0x7f17d4102a60 2024-09-21T05:56:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:56:43.698 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"btime":"2024-09-21T05:56:39:464552+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:56:39.280658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:56:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 54 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 msgr2=0x7f17c00b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 0x7f17c00b8bc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f17d4105f80 tx=0x7f17c4005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 msgr2=0x7f17d419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f17d000eb90 tx=0x7f17d000c6a0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 shutdown_connections 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17d4108cf0 0x7f17d419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17d41083a0 0x7f17d419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17c00b66b0 0x7f17c00b8bc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 --2- 172.21.15.28:0/4124513171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102a60 0x7f17d419b290 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 >> 172.21.15.28:0/4124513171 conn(0x7f17d4079730 msgr2=0x7f17d4104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 shutdown_connections 2024-09-21T05:56:43.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:43.698+0000 7f17dbd83640 1 -- 172.21.15.28:0/4124513171 wait complete. 2024-09-21T05:56:44.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:4 damage ls 2024-09-21T05:56:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:44.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:44 smithi181 ceph-mon[34819]: pgmap v621: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T05:56:44.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:44 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/4124513171' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:56:44.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:44 smithi028 ceph-mon[31227]: pgmap v621: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T05:56:44.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:44 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/4124513171' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:56:44.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:44 smithi129 ceph-mon[34885]: pgmap v621: 97 pgs: 17 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 46 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T05:56:44.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:44 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/4124513171' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:56:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 -- 172.21.15.28:0/4262349017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 msgr2=0x7f5d70102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 --2- 172.21.15.28:0/4262349017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d70102c80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009900 tx=0x7f5d6c02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 -- 172.21.15.28:0/4262349017 shutdown_connections 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 --2- 172.21.15.28:0/4262349017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d70108b00 0x7f5d7010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 --2- 172.21.15.28:0/4262349017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d701081b0 0x7f5d701085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 --2- 172.21.15.28:0/4262349017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d70102c80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 -- 172.21.15.28:0/4262349017 >> 172.21.15.28:0/4262349017 conn(0x7f5d700fe2c0 msgr2=0x7f5d70100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 -- 172.21.15.28:0/4262349017 shutdown_connections 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.225+0000 7f5d7639f640 1 -- 172.21.15.28:0/4262349017 wait complete. 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 Processor -- start 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 -- start start 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d701081b0 0x7f5d7019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d70108b00 0x7f5d701a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701138d0 con 0x7f5d70102870 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70113750 con 0x7f5d70108b00 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7639f640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70113a50 con 0x7f5d701081b0 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7539d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.226+0000 7f5d7539d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57474/0 (socket says 172.21.15.28:57474) 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 -- 172.21.15.28:0/591732401 learned_addr learned my addr 172.21.15.28:0/591732401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d701081b0 0x7f5d7019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d70108b00 0x7f5d701a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 -- 172.21.15.28:0/591732401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d701081b0 msgr2=0x7f5d7019ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d701081b0 0x7f5d7019ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 -- 172.21.15.28:0/591732401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d70108b00 msgr2=0x7f5d701a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d70108b00 0x7f5d701a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 -- 172.21.15.28:0/591732401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d701a2430 con 0x7f5d70102870 2024-09-21T05:56:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7539d640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005940 tx=0x7f5d6c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d667fc640 1 -- 172.21.15.28:0/591732401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c03f030 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d667fc640 1 -- 172.21.15.28:0/591732401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c0044c0 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d701a26c0 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d667fc640 1 -- 172.21.15.28:0/591732401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c035070 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.227+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701a8e80 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.230+0000 7f5d7db75740 1 Processor -- start 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.230+0000 7f5d7db75740 1 -- start start 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.230+0000 7f5d667fc640 1 -- 172.21.15.28:0/591732401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f5d6c042030 con 0x7f5d70102870 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a88da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66c0390 0x562fb6a892e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fb69feea0 con 0x562fb66b81b0 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562fb69ee2d0 con 0x562fb66c0390 2024-09-21T05:56:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7db75740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562fb69f20e0 con 0x562fb66e1070 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7539d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a88da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7539d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a88da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57484/0 (socket says 172.21.15.28:57484) 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d7539d640 1 -- 172.21.15.28:0/2672149808 learned_addr learned my addr 172.21.15.28:0/2672149808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:52936/0 (socket says 172.21.15.28:52936) 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66c0390 0x562fb6a892e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2672149808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66c0390 msgr2=0x562fb6a892e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66c0390 0x562fb6a892e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2672149808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 msgr2=0x562fb6a88da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a88da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2672149808 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fb674c650 con 0x562fb66e1070 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009ed0 tx=0x7f5d680107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e4e3bad4b59ee3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.231+0000 7f5d667fc640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 0x7f5d5c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.232+0000 7f5d5b7fe640 1 -- 172.21.15.28:0/2672149808 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68003b70 con 0x562fb66e1070 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.232+0000 7f5d5b7fe640 1 -- 172.21.15.28:0/2672149808 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d68015070 con 0x562fb66e1070 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.232+0000 7f5d7539d640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 0x7f5d5c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.232+0000 7f5d5b7fe640 1 -- 172.21.15.28:0/2672149808 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68011750 con 0x562fb66e1070 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.232+0000 7f5d667fc640 1 -- 172.21.15.28:0/591732401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f5d6c107b30 con 0x7f5d70102870 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- 172.21.15.28:0/2672149808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 msgr2=0x562fb6a88860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009ed0 tx=0x7f5d680107a0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7539d640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 0x7f5d5c0b8b90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0c9f20 tx=0x7f5d6c10b680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- 172.21.15.28:0/2672149808 shutdown_connections 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66c0390 0x562fb6a892e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a88da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2672149808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66e1070 0x562fb6a88860 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- 172.21.15.28:0/2672149808 >> 172.21.15.28:0/2672149808 conn(0x562fb66d5460 msgr2=0x562fb6a86750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- 172.21.15.28:0/2672149808 shutdown_connections 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- 172.21.15.28:0/2672149808 wait complete. 2024-09-21T05:56:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 -- start start 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.233+0000 7f5d7db75740 1 Processor -- start 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a93f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a94440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fb684f700 con 0x562fb66e1070 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562fb6989160 con 0x562fb66b81b0 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562fb6a97040 con 0x562fb66c0390 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d7db75740 10 client.0 _ll_register_callbacks cb 0x562fb69a29e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d74b9c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a94440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57498/0 (socket says 172.21.15.28:57498) 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2300347880 learned_addr learned my addr 172.21.15.28:0/2300347880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a93f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2300347880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 msgr2=0x562fb6a94440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a94440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 msgr2=0x562fb6a93f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a93f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 -- 172.21.15.28:0/2300347880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fb69b6c90 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.234+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5d680047a0 tx=0x7f5d68004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac117714ad3aea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68020070 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f5d68003c80 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d68015070 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5d68031050 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 10 client.17769 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x562fb6aa2d40 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d68015070 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f5d680118e0 con 0x562fb66e1070 2024-09-21T05:56:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 10 client.17769 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 10 client.17769 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.235+0000 7f5d7db75740 20 client.17769 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb669f9f0 0x562fb6aa61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d7db75740 4 client.17769 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fb6a00df0 con 0x562fb669f9f0 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb669f9f0 0x562fb6aa61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb669f9f0 0x562fb6aa61b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=495260d9d9609d2d server_cookie=e997828c8d9d63db in_seq=0 out_seq=0 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.236+0000 7f5d58ff9640 10 client.17769 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d58ff9640 1 -- 172.21.15.28:0/2300347880 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562fb6a00df0 con 0x562fb669f9f0 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d58ff9640 10 client.17769 handle_command_reply: tid=0 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 client.17769 shutdown 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 msgr2=0x562fb6a949c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5d680047a0 tx=0x7f5d68004940 comp rx=0 tx=0).stop 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 shutdown_connections 2024-09-21T05:56:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb669f9f0 0x562fb6aa61b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a949c0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a94440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 --2- 172.21.15.28:0/2300347880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a93f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 >> 172.21.15.28:0/2300347880 conn(0x562fb66d5460 msgr2=0x562fb6a8b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d58ff9640 0 client.17769 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 shutdown_connections 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 1 -- 172.21.15.28:0/2300347880 wait complete. 2024-09-21T05:56:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.237+0000 7f5d7db75740 20 client.17769 trim_cache size 0 max 16384 2024-09-21T05:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 Processor -- start 2024-09-21T05:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 -- start start 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a85d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66e1070 0x562fb6a86260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701b17b0 con 0x562fb66b81b0 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701b32f0 con 0x562fb66e1070 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d7db75740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701b1e80 con 0x562fb66c0390 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57508/0 (socket says 172.21.15.28:57508) 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d75b9e640 1 -- 172.21.15.28:0/1109929054 learned_addr learned my addr 172.21.15.28:0/1109929054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.251+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a85d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 -- 172.21.15.28:0/1109929054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 msgr2=0x562fb6a85d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a85d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 -- 172.21.15.28:0/1109929054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66e1070 msgr2=0x562fb6a86260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66e1070 0x562fb6a86260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 -- 172.21.15.28:0/1109929054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fb6a8b0a0 con 0x562fb66b81b0 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009a40 tx=0x7f5d68011510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5868ce73aff26ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d5affd640 1 -- 172.21.15.28:0/1109929054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68020070 con 0x562fb66b81b0 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d5affd640 1 -- 172.21.15.28:0/1109929054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d68015040 con 0x562fb66b81b0 2024-09-21T05:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.252+0000 7f5d5affd640 1 -- 172.21.15.28:0/1109929054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68011a30 con 0x562fb66b81b0 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 -- 172.21.15.28:0/1109929054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 msgr2=0x562fb6a95ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009a40 tx=0x7f5d68011510 comp rx=0 tx=0).stop 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 -- 172.21.15.28:0/1109929054 shutdown_connections 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66e1070 0x562fb6a86260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a85d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 --2- 172.21.15.28:0/1109929054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66b81b0 0x562fb6a95ef0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 -- 172.21.15.28:0/1109929054 >> 172.21.15.28:0/1109929054 conn(0x562fb66d5460 msgr2=0x562fb6a9fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 -- 172.21.15.28:0/1109929054 shutdown_connections 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.253+0000 7f5d7db75740 1 -- 172.21.15.28:0/1109929054 wait complete. 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 -- start start 2024-09-21T05:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 Processor -- start 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a98500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a98a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a98fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701b2c20 con 0x562fb66e1070 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701b2550 con 0x562fb66b81b0 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701af7b0 con 0x562fb66c0390 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d7db75740 10 client.0 _ll_register_callbacks cb 0x7f5d6c10b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a98fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d74b9c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a98a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d75b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a98fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57512/0 (socket says 172.21.15.28:57512) 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.254+0000 7f5d75b9e640 1 -- 172.21.15.28:0/3518500025 learned_addr learned my addr 172.21.15.28:0/3518500025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a98500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 -- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 msgr2=0x562fb6a98500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a98500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 -- 172.21.15.28:0/3518500025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 msgr2=0x562fb6a98fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a98fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 -- 172.21.15.28:0/3518500025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fb6aa6610 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d74b9c640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a98a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x562fb6aa67e0 tx=0x7f5d600109b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e750f2894d3a88 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6001f460 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d68015040 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f5d6001f600 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5d60031050 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d7db75740 10 client.27244 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.255+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x562fb6b10e00 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d68015040 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f5d60023070 con 0x562fb66c0390 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 10 client.27244 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 10 client.27244 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-09-21T05:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 20 client.27244 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb6674da0 0x562fb6b135e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 4 client.27244 mds_command: new command op to 24653 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 --> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fb69fc8a0 con 0x562fb6674da0 2024-09-21T05:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.256+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb6674da0 0x562fb6b135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d58ff9640 10 client.27244 ms_handle_connect on v2:172.21.15.129:6836/2283221342 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d75b9e640 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb6674da0 0x562fb6b135e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eedc607ad1169bb4 server_cookie=940699b8aa943c36 in_seq=0 out_seq=0 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d58ff9640 1 -- 172.21.15.28:0/3518500025 <== mds.4 v2:172.21.15.129:6836/2283221342 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fb69fc8a0 con 0x562fb6674da0 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d58ff9640 10 client.27244 handle_command_reply: tid=0 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 client.27244 shutdown 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 msgr2=0x562fb6a98a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a98a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x562fb6aa67e0 tx=0x7f5d600109b0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 shutdown_connections 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:6836/2283221342,v1:172.21.15.129:6837/2283221342] conn(0x562fb6674da0 0x562fb6b135e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fb66e1070 0x562fb6a98fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562fb66c0390 0x562fb6a98a40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d58ff9640 0 client.27244 ms_handle_reset on v2:172.21.15.129:6836/2283221342 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 --2- 172.21.15.28:0/3518500025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562fb66b81b0 0x562fb6a98500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 >> 172.21.15.28:0/3518500025 conn(0x562fb66d5460 msgr2=0x562fb6a92780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 shutdown_connections 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.257+0000 7f5d7db75740 1 -- 172.21.15.28:0/3518500025 wait complete. 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7db75740 20 client.27244 trim_cache size 0 max 16384 2024-09-21T05:56:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 msgr2=0x7f5d5c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 0x7f5d5c0b8b90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0c9f20 tx=0x7f5d6c10b680 comp rx=0 tx=0).stop 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 msgr2=0x7f5d7019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005940 tx=0x7f5d6c0040c0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 shutdown_connections 2024-09-21T05:56:45.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d70108b00 0x7f5d701a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d701081b0 0x7f5d7019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f5d5c0b6680 0x7f5d5c0b8b90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 --2- 172.21.15.28:0/591732401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d70102870 0x7f5d7019c4c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 >> 172.21.15.28:0/591732401 conn(0x7f5d700fe2c0 msgr2=0x7f5d70107870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.258+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 shutdown_connections 2024-09-21T05:56:45.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:45.259+0000 7f5d7639f640 1 -- 172.21.15.28:0/591732401 wait complete. 2024-09-21T05:56:46.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:3 damage ls 2024-09-21T05:56:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:46.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:46 smithi028 ceph-mon[31227]: pgmap v622: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:46.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:46 smithi181 ceph-mon[34819]: pgmap v622: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.784+0000 7f609c445640 1 -- 172.21.15.28:0/2033864074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 msgr2=0x7f60941098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.784+0000 7f609c445640 1 --2- 172.21.15.28:0/2033864074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f60941098b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6088009900 tx=0x7f608802a3f0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.784+0000 7f609c445640 1 -- 172.21.15.28:0/2033864074 shutdown_connections 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.784+0000 7f609c445640 1 --2- 172.21.15.28:0/2033864074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 0x7f6094110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 --2- 172.21.15.28:0/2033864074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f60941098b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 --2- 172.21.15.28:0/2033864074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f6094103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 -- 172.21.15.28:0/2033864074 >> 172.21.15.28:0/2033864074 conn(0x7f609407b850 msgr2=0x7f609407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 -- 172.21.15.28:0/2033864074 shutdown_connections 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 -- 172.21.15.28:0/2033864074 wait complete. 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 Processor -- start 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.785+0000 7f609c445640 1 -- start start 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f609419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 0x7f609419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6094113ab0 con 0x7f60941094a0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6094113930 con 0x7f6094103b60 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609c445640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6094113c30 con 0x7f6094109df0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57532/0 (socket says 172.21.15.28:57532) 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 -- 172.21.15.28:0/3205047211 learned_addr learned my addr 172.21.15.28:0/3205047211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609a1e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f609419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609a9e2640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 0x7f609419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 -- 172.21.15.28:0/3205047211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 msgr2=0x7f609419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 0x7f609419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 -- 172.21.15.28:0/3205047211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 msgr2=0x7f609419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f609419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 -- 172.21.15.28:0/3205047211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60941a0430 con 0x7f60941094a0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f609a1e1640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f609419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.786+0000 7f60999e0640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f608802a910 tx=0x7f6088004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.787+0000 7f60937fe640 1 -- 172.21.15.28:0/3205047211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608803f030 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.787+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60941a06c0 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.787+0000 7f60937fe640 1 -- 172.21.15.28:0/3205047211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6088003aa0 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.787+0000 7f60937fe640 1 -- 172.21.15.28:0/3205047211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088035070 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.787+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60941a0bf0 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.789+0000 7f60a29b9740 1 Processor -- start 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 -- start start 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60937fe640 1 -- 172.21.15.28:0/3205047211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f6088042030 con 0x7f60941094a0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa85070 0x56287ae2c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae2cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56287ada2ea0 con 0x56287aa64390 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56287ad922d0 con 0x56287aa85070 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60a29b9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56287ad960e0 con 0x56287aa5c1b0 2024-09-21T05:56:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae2cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:52984/0 (socket says 172.21.15.28:52984) 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 -- 172.21.15.28:0/2652586696 learned_addr learned my addr 172.21.15.28:0/2652586696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a9e2640 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa85070 0x56287ae2c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 -- 172.21.15.28:0/2652586696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa85070 msgr2=0x56287ae2c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa85070 0x56287ae2c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 -- 172.21.15.28:0/2652586696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 msgr2=0x56287ae2cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae2cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.790+0000 7f609a1e1640 1 -- 172.21.15.28:0/2652586696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56287aaf0650 con 0x56287aa5c1b0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f609a1e1640 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6084010860 tx=0x7f6084010d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d45ecf2e67701945 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f60937fe640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 0x7f607c0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f609a9e2640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 0x7f607c0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f6090d9a640 1 -- 172.21.15.28:0/2652586696 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084004d20 con 0x56287aa5c1b0 2024-09-21T05:56:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f6090d9a640 1 -- 172.21.15.28:0/2652586696 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6084015070 con 0x56287aa5c1b0 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f60937fe640 1 -- 172.21.15.28:0/3205047211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f6088107e00 con 0x7f60941094a0 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.791+0000 7f6090d9a640 1 -- 172.21.15.28:0/2652586696 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60840057b0 con 0x56287aa5c1b0 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- 172.21.15.28:0/2652586696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 msgr2=0x56287ae2c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f609a9e2640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 0x7f607c0b8bc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f608c00ac10 tx=0x7f608c00a250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6084010860 tx=0x7f6084010d20 comp rx=0 tx=0).stop 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- 172.21.15.28:0/2652586696 shutdown_connections 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae2cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa5c1b0 0x56287ae2c800 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 --2- 172.21.15.28:0/2652586696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa85070 0x56287ae2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- 172.21.15.28:0/2652586696 >> 172.21.15.28:0/2652586696 conn(0x56287aa79460 msgr2=0x56287ae2a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- 172.21.15.28:0/2652586696 shutdown_connections 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- 172.21.15.28:0/2652586696 wait complete. 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.792+0000 7f60a29b9740 1 -- start start 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 Processor -- start 2024-09-21T05:56:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae48130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56287abf3700 con 0x56287aa64390 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56287ad2d160 con 0x56287aa5c1b0 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56287ae3ee00 con 0x56287aa85070 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60a29b9740 10 client.0 _ll_register_callbacks cb 0x56287ad469e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57554/0 (socket says 172.21.15.28:57554) 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 -- 172.21.15.28:0/3752209764 learned_addr learned my addr 172.21.15.28:0/3752209764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f609a1e1640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f609a9e2640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae48130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 -- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 msgr2=0x56287ae48130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae48130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 -- 172.21.15.28:0/3752209764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 msgr2=0x56287ae3bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.793+0000 7f60999e0640 1 -- 172.21.15.28:0/3752209764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56287ad5ac90 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60999e0640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6088105580 tx=0x7f6088000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f1d26e2e98e81b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088049070 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f60881057a0 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6084015070 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f60880c9b70 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 10 client.17787 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x56287ae4ab80 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6084015070 con 0x56287aa64390 2024-09-21T05:56:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f60880c9f40 con 0x56287aa64390 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 10 client.17787 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 10 client.17787 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 20 client.17787 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa439f0 0x56287ae4e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 4 client.17787 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.794+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56287ada4df0 con 0x56287aa439f0 2024-09-21T05:56:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.795+0000 7f609a1e1640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa439f0 0x56287ae4e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.795+0000 7f6079ffb640 10 client.17787 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.795+0000 7f609a1e1640 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa439f0 0x56287ae4e110 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9238d7f9daf5bdcc server_cookie=9f562d9da84378a8 in_seq=0 out_seq=0 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f6079ffb640 1 -- 172.21.15.28:0/3752209764 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56287ada4df0 con 0x56287aa439f0 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f6079ffb640 10 client.17787 handle_command_reply: tid=0 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 client.17787 shutdown 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 msgr2=0x56287ae3c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6088105580 tx=0x7f6088000bc0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 shutdown_connections 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa439f0 0x56287ae4e110 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae48130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3c320 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 --2- 172.21.15.28:0/3752209764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 >> 172.21.15.28:0/3752209764 conn(0x56287aa79460 msgr2=0x56287ae2dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f6079ffb640 0 client.17787 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 shutdown_connections 2024-09-21T05:56:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 1 -- 172.21.15.28:0/3752209764 wait complete. 2024-09-21T05:56:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.796+0000 7f60a29b9740 20 client.17787 trim_cache size 0 max 16384 2024-09-21T05:56:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 Processor -- start 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 -- start start 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae3bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60941ad6a0 con 0x56287aa64390 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60941af1e0 con 0x56287aa5c1b0 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.811+0000 7f60a29b9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f60941add70 con 0x56287aa85070 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f609a1e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57570/0 (socket says 172.21.15.28:57570) 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 -- 172.21.15.28:0/722606251 learned_addr learned my addr 172.21.15.28:0/722606251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f609a9e2640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae3bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 -- 172.21.15.28:0/722606251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 msgr2=0x56287ae3bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae3bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 -- 172.21.15.28:0/722606251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 msgr2=0x56287ae3a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 -- 172.21.15.28:0/722606251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56287aa43c20 con 0x56287aa64390 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f609a1e1640 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f60999e0640 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f60880cd9d0 tx=0x7f608803b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d1a01de78b7302 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f607bfff640 1 -- 172.21.15.28:0/722606251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088049020 con 0x56287aa64390 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f607bfff640 1 -- 172.21.15.28:0/722606251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608810b640 con 0x56287aa64390 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.812+0000 7f607bfff640 1 -- 172.21.15.28:0/722606251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60880c9440 con 0x56287aa64390 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 -- 172.21.15.28:0/722606251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 msgr2=0x56287ae3aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f60880cd9d0 tx=0x7f608803b6b0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 -- 172.21.15.28:0/722606251 shutdown_connections 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae3bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3aba0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 --2- 172.21.15.28:0/722606251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae3a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 -- 172.21.15.28:0/722606251 >> 172.21.15.28:0/722606251 conn(0x56287aa79460 msgr2=0x56287ae31c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 -- 172.21.15.28:0/722606251 shutdown_connections 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.813+0000 7f60a29b9740 1 -- 172.21.15.28:0/722606251 wait complete. 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 -- start start 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 Processor -- start 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60941aeb10 con 0x56287aa64390 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60941ae440 con 0x56287aa5c1b0 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f60941ab6a0 con 0x56287aa85070 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60a29b9740 10 client.0 _ll_register_callbacks cb 0x7f6088005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60999e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57582/0 (socket says 172.21.15.28:57582) 2024-09-21T05:56:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f60999e0640 1 -- 172.21.15.28:0/1231298659 learned_addr learned my addr 172.21.15.28:0/1231298659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.814+0000 7f609a9e2640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f609a1e1640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 -- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 msgr2=0x56287ae402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 -- 172.21.15.28:0/1231298659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 msgr2=0x56287ae448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 -- 172.21.15.28:0/1231298659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56287ae40a40 con 0x56287aa64390 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f609a9e2640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60999e0640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f60880b52c0 tx=0x7f6088009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373b772a652bedbd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f6079ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088049030 con 0x56287aa64390 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f60880c9f40 con 0x56287aa64390 2024-09-21T05:56:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f608810b640 con 0x56287aa64390 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f60880cd440 con 0x56287aa64390 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60a29b9740 10 client.17793 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x56287aeb8a30 con 0x56287aa64390 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.815+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608810b640 con 0x56287aa64390 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f608810bb00 con 0x56287aa64390 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 10 client.17793 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 10 client.17793 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 20 client.17793 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa18da0 0x56287aebb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 4 client.17793 mds_command: new command op to 24664 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 --> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56287ada08a0 con 0x56287aa18da0 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f609a1e1640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa18da0 0x56287aebb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f609a1e1640 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa18da0 0x56287aebb050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e4be0de76033952 server_cookie=5d8cf67bace21a67 in_seq=0 out_seq=0 2024-09-21T05:56:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.816+0000 7f6079ffb640 10 client.17793 ms_handle_connect on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f6079ffb640 1 -- 172.21.15.28:0/1231298659 <== mds.3 v2:172.21.15.181:6834/1316004847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56287ada08a0 con 0x56287aa18da0 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f6079ffb640 10 client.17793 handle_command_reply: tid=0 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 client.17793 shutdown 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 msgr2=0x56287ae3fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f60880b52c0 tx=0x7f6088009260 comp rx=0 tx=0).stop 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 shutdown_connections 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:6834/1316004847,v1:172.21.15.181:6835/1316004847] conn(0x56287aa18da0 0x56287aebb050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56287aa85070 0x56287ae402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56287aa64390 0x56287ae3fd50 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 --2- 172.21.15.28:0/1231298659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56287aa5c1b0 0x56287ae448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f6079ffb640 0 client.17793 ms_handle_reset on v2:172.21.15.181:6834/1316004847 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 >> 172.21.15.28:0/1231298659 conn(0x56287aa79460 msgr2=0x56287ae3a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 shutdown_connections 2024-09-21T05:56:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 1 -- 172.21.15.28:0/1231298659 wait complete. 2024-09-21T05:56:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f60a29b9740 20 client.17793 trim_cache size 0 max 16384 2024-09-21T05:56:46.825 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 msgr2=0x7f607c0b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 0x7f607c0b8bc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f608c00ac10 tx=0x7f608c00a250 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 msgr2=0x7f609419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.817+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f608802a910 tx=0x7f6088004270 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 shutdown_connections 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6094109df0 0x7f609419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f607c0b66b0 0x7f607c0b8bc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60941094a0 0x7f609419ab20 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 --2- 172.21.15.28:0/3205047211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6094103b60 0x7f609419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 >> 172.21.15.28:0/3205047211 conn(0x7f609407b850 msgr2=0x7f6094105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 shutdown_connections 2024-09-21T05:56:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:46.818+0000 7f609c445640 1 -- 172.21.15.28:0/3205047211 wait complete. 2024-09-21T05:56:46.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:46 smithi129 ceph-mon[34885]: pgmap v622: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:47.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:2 damage ls 2024-09-21T05:56:47.910 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 -- 172.21.15.28:0/4206925960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0108d20 msgr2=0x7fe3f010f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 --2- 172.21.15.28:0/4206925960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0108d20 0x7fe3f010f550 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009ca0 tx=0x7fe3e802a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 -- 172.21.15.28:0/4206925960 shutdown_connections 2024-09-21T05:56:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 --2- 172.21.15.28:0/4206925960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0108d20 0x7fe3f010f550 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 --2- 172.21.15.28:0/4206925960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f01083d0 0x7fe3f01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 --2- 172.21.15.28:0/4206925960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f0102a90 0x7fe3f0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.429+0000 7fe3f5006640 1 -- 172.21.15.28:0/4206925960 >> 172.21.15.28:0/4206925960 conn(0x7fe3f0079750 msgr2=0x7fe3f0079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.430+0000 7fe3f5006640 1 -- 172.21.15.28:0/4206925960 shutdown_connections 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.430+0000 7fe3f5006640 1 -- 172.21.15.28:0/4206925960 wait complete. 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 Processor -- start 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 -- start start 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 0x7fe3f019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f0108d20 0x7fe3f019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0112990 con 0x7fe3f0102a90 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0112810 con 0x7fe3f0108d20 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3f5006640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0112b10 con 0x7fe3f01083d0 2024-09-21T05:56:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57600/0 (socket says 172.21.15.28:57600) 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3eed9d640 1 -- 172.21.15.28:0/2506569139 learned_addr learned my addr 172.21.15.28:0/2506569139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 0x7fe3f019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.431+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f0108d20 0x7fe3f019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 -- 172.21.15.28:0/2506569139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f0108d20 msgr2=0x7fe3f019bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f0108d20 0x7fe3f019bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 -- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 msgr2=0x7fe3f019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 -- 172.21.15.28:0/2506569139 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f01990f0 con 0x7fe3f01083d0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 0x7fe3f019b7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8005fd0 tx=0x7fe3d800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3e7fff640 1 -- 172.21.15.28:0/2506569139 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8009c00 con 0x7fe3f01083d0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f01993e0 con 0x7fe3f01083d0 2024-09-21T05:56:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3e7fff640 1 -- 172.21.15.28:0/2506569139 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3d8009da0 con 0x7fe3f01083d0 2024-09-21T05:56:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3e7fff640 1 -- 172.21.15.28:0/2506569139 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8003e40 con 0x7fe3f01083d0 2024-09-21T05:56:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.432+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f0199970 con 0x7fe3f01083d0 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 Processor -- start 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 -- start start 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3e7fff640 1 -- 172.21.15.28:0/2506569139 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fe3d8010460 con 0x7fe3f01083d0 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fcbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d00f41ea0 con 0x557d00c24070 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d00f312d0 con 0x557d00c03390 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d00f350e0 con 0x557d00bfb1b0 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.435+0000 7fe3ef59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ee59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57606/0 (socket says 172.21.15.28:57606) 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 -- 172.21.15.28:0/3268539409 learned_addr learned my addr 172.21.15.28:0/3268539409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fcbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 -- 172.21.15.28:0/3268539409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 msgr2=0x557d00fcbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fcbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 -- 172.21.15.28:0/3268539409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 msgr2=0x557d00fcc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 -- 172.21.15.28:0/3268539409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d00c8f650 con 0x557d00c24070 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8035000 tx=0x7fe3e802f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc383da54fd0e44f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3e57fa640 1 -- 172.21.15.28:0/3268539409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8031cf0 con 0x557d00c24070 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3e57fa640 1 -- 172.21.15.28:0/3268539409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e8036030 con 0x557d00c24070 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.436+0000 7fe3e57fa640 1 -- 172.21.15.28:0/3268539409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8004280 con 0x557d00c24070 2024-09-21T05:56:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3e7fff640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 0x7fe3dc0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 0x7fe3dc0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3e7fff640 1 -- 172.21.15.28:0/2506569139 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7fe3d80e7130 con 0x7fe3f01083d0 2024-09-21T05:56:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3fb57a740 1 -- 172.21.15.28:0/3268539409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 msgr2=0x557d00fcb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 0x7fe3dc0b8b90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x557d00fca320 tx=0x7fe3e0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.437+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8035000 tx=0x7fe3e802f4c0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 -- 172.21.15.28:0/3268539409 shutdown_connections 2024-09-21T05:56:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fcbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/3268539409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c24070 0x557d00fcb6a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 -- 172.21.15.28:0/3268539409 >> 172.21.15.28:0/3268539409 conn(0x557d00c18460 msgr2=0x557d00fc9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 -- 172.21.15.28:0/3268539409 shutdown_connections 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 -- 172.21.15.28:0/3268539409 wait complete. 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 -- start start 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 Processor -- start 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.438+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 0x557d00fd7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fd7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d00d92700 con 0x557d00bfb1b0 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557d00ecc160 con 0x557d00c24070 2024-09-21T05:56:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557d00fd9ca0 con 0x557d00c03390 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57608/0 (socket says 172.21.15.28:57608) 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3eed9d640 1 -- 172.21.15.28:0/515825149 learned_addr learned my addr 172.21.15.28:0/515825149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fd7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3fb57a740 10 client.0 _ll_register_callbacks cb 0x557d00ee59e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 0x557d00fd7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 -- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 msgr2=0x557d00fd7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 0x557d00fd7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 -- 172.21.15.28:0/515825149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 msgr2=0x557d00fd6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 -- 172.21.15.28:0/515825149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d00ef9c90 con 0x557d00c24070 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 0x557d00fd7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fd7790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe3e802ade0 tx=0x7fe3e80099f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5299b5c288aca34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.439+0000 7fe3d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e804a040 con 0x557d00c24070 2024-09-21T05:56:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e8036030 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7fe3e8031650 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe3e805e050 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 10 client.27290 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557d00fe5900 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e8036030 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7fe3e804d030 con 0x557d00c24070 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 10 client.27290 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 10 client.27290 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 20 client.27290 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00be29f0 0x557d00fe8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 4 client.27290 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.440+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d00f43df0 con 0x557d00be29f0 2024-09-21T05:56:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.441+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00be29f0 0x557d00fe8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.441+0000 7fe3d67fc640 10 client.27290 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.441+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00be29f0 0x557d00fe8c60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=741a755319b26aa server_cookie=599d0acdc3edec11 in_seq=0 out_seq=0 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3d67fc640 1 -- 172.21.15.28:0/515825149 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557d00f43df0 con 0x557d00be29f0 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3d67fc640 10 client.27290 handle_command_reply: tid=0 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 client.27290 shutdown 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 msgr2=0x557d00fd7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fd7790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe3e802ade0 tx=0x7fe3e80099f0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 shutdown_connections 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00be29f0 0x557d00fe8c60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fd7790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c03390 0x557d00fd7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3d67fc640 0 client.27290 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/515825149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 >> 172.21.15.28:0/515825149 conn(0x557d00c18460 msgr2=0x557d00fce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 shutdown_connections 2024-09-21T05:56:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 1 -- 172.21.15.28:0/515825149 wait complete. 2024-09-21T05:56:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.442+0000 7fe3fb57a740 20 client.27290 trim_cache size 0 max 16384 2024-09-21T05:56:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.455+0000 7fe3fb57a740 1 Processor -- start 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.455+0000 7fe3fb57a740 1 -- start start 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c24070 0x557d00fe2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b18c0 con 0x557d00bfb1b0 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b3400 con 0x557d00c03390 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b1f90 con 0x557d00c24070 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57624/0 (socket says 172.21.15.28:57624) 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 -- 172.21.15.28:0/227629243 learned_addr learned my addr 172.21.15.28:0/227629243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c24070 0x557d00fe2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 -- 172.21.15.28:0/227629243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c24070 msgr2=0x557d00fe2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c24070 0x557d00fe2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 -- 172.21.15.28:0/227629243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 msgr2=0x557d00fcc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 -- 172.21.15.28:0/227629243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d00fce500 con 0x557d00bfb1b0 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0012fe0 tx=0x7fe3e001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6149dcb2eddbcf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.456+0000 7fe3e4ff9640 1 -- 172.21.15.28:0/227629243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0004490 con 0x557d00bfb1b0 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.457+0000 7fe3e4ff9640 1 -- 172.21.15.28:0/227629243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e000fb90 con 0x557d00bfb1b0 2024-09-21T05:56:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.457+0000 7fe3e4ff9640 1 -- 172.21.15.28:0/227629243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e002b690 con 0x557d00bfb1b0 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.457+0000 7fe3fb57a740 1 -- 172.21.15.28:0/227629243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 msgr2=0x557d00fd8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.457+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0012fe0 tx=0x7fe3e001a000 comp rx=0 tx=0).stop 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.457+0000 7fe3fb57a740 1 -- 172.21.15.28:0/227629243 shutdown_connections 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00c24070 0x557d00fe2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c03390 0x557d00fcc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/227629243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00bfb1b0 0x557d00fd8bb0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- 172.21.15.28:0/227629243 >> 172.21.15.28:0/227629243 conn(0x557d00c18460 msgr2=0x557d00fe1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- 172.21.15.28:0/227629243 shutdown_connections 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- 172.21.15.28:0/227629243 wait complete. 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- start start 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 Processor -- start 2024-09-21T05:56:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 0x557d00fdac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fdb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b2d30 con 0x557d00c03390 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b2660 con 0x557d00c24070 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01af8c0 con 0x557d00bfb1b0 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.458+0000 7fe3fb57a740 10 client.0 _ll_register_callbacks cb 0x7fe3d80eabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3eed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:53052/0 (socket says 172.21.15.28:53052) 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fdb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3eed9d640 1 -- 172.21.15.28:0/1330711834 learned_addr learned my addr 172.21.15.28:0/1330711834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ee59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 0x557d00fdac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 -- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 msgr2=0x557d00fdf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 -- 172.21.15.28:0/1330711834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 msgr2=0x557d00fdac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 0x557d00fdac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 -- 172.21.15.28:0/1330711834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d00fdba40 con 0x557d00c24070 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ee59c640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 0x557d00fdac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3ef59e640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fdb2b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009a60 tx=0x7fe3e8009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac9a6212e4ff6f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8031450 con 0x557d00c24070 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7fe3e80315f0 con 0x557d00c24070 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.459+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e000fb90 con 0x557d00c24070 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe3e8031830 con 0x557d00c24070 2024-09-21T05:56:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 10 client.27296 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557d01053530 con 0x557d00c24070 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e000fb90 con 0x557d00c24070 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7fe3e804d030 con 0x557d00c24070 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 10 client.27296 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 10 client.27296 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 20 client.27296 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00bb7da0 0x557d01055cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 4 client.27296 mds_command: new command op to 24673 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 --> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d00f3f8a0 con 0x557d00bb7da0 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.460+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00bb7da0 0x557d01055cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3eed9d640 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00bb7da0 0x557d01055cf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28337eaff8b75b69 server_cookie=37c849f39dbdb767 in_seq=0 out_seq=0 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3d67fc640 10 client.27296 ms_handle_connect on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3d67fc640 1 -- 172.21.15.28:0/1330711834 <== mds.2 v2:172.21.15.181:6836/2388771333 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d00f3f8a0 con 0x557d00bb7da0 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3d67fc640 10 client.27296 handle_command_reply: tid=0 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 client.27296 shutdown 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 msgr2=0x557d00fdb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fdb2b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009a60 tx=0x7fe3e8009630 comp rx=0 tx=0).stop 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 shutdown_connections 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:6836/2388771333,v1:172.21.15.181:6837/2388771333] conn(0x557d00bb7da0 0x557d01055cf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557d00c24070 0x557d00fdb2b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d00c03390 0x557d00fdac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 --2- 172.21.15.28:0/1330711834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d00bfb1b0 0x557d00fdf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3d67fc640 0 client.27296 ms_handle_reset on v2:172.21.15.181:6836/2388771333 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 >> 172.21.15.28:0/1330711834 conn(0x557d00c18460 msgr2=0x557d00fe1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 shutdown_connections 2024-09-21T05:56:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 1 -- 172.21.15.28:0/1330711834 wait complete. 2024-09-21T05:56:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.461+0000 7fe3fb57a740 20 client.27296 trim_cache size 0 max 16384 2024-09-21T05:56:48.469 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 msgr2=0x7fe3dc0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 0x7fe3dc0b8b90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x557d00fca320 tx=0x7fe3e0005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 msgr2=0x7fe3f019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 0x7fe3f019b7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8005fd0 tx=0x7fe3d800dde0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 shutdown_connections 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3f0108d20 0x7fe3f019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f01083d0 0x7fe3f019b7c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fe3dc0b6680 0x7fe3dc0b8b90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 --2- 172.21.15.28:0/2506569139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0102a90 0x7fe3f019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 >> 172.21.15.28:0/2506569139 conn(0x7fe3f0079750 msgr2=0x7fe3f01044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 shutdown_connections 2024-09-21T05:56:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:48.462+0000 7fe3f5006640 1 -- 172.21.15.28:0/2506569139 wait complete. 2024-09-21T05:56:48.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:48 smithi028 ceph-mon[31227]: pgmap v623: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:48.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:48 smithi181 ceph-mon[34819]: pgmap v623: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:48.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:48 smithi129 ceph-mon[34885]: pgmap v623: 97 pgs: 16 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 51 active+clean; 17 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:49.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:1 damage ls 2024-09-21T05:56:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 -- 172.21.15.28:0/1163161025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770102ae0 msgr2=0x7f1770102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 --2- 172.21.15.28:0/1163161025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770102ae0 0x7f1770102ef0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1760009900 tx=0x7f176002a410 comp rx=0 tx=0).stop 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 -- 172.21.15.28:0/1163161025 shutdown_connections 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 --2- 172.21.15.28:0/1163161025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 0x7f177010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 --2- 172.21.15.28:0/1163161025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17701083d0 0x7f17701087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 --2- 172.21.15.28:0/1163161025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1770102ae0 0x7f1770102ef0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 -- 172.21.15.28:0/1163161025 >> 172.21.15.28:0/1163161025 conn(0x7f1770079760 msgr2=0x7f1770079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 -- 172.21.15.28:0/1163161025 shutdown_connections 2024-09-21T05:56:50.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.014+0000 7f177491b640 1 -- 172.21.15.28:0/1163161025 wait complete. 2024-09-21T05:56:50.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 Processor -- start 2024-09-21T05:56:50.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 -- start start 2024-09-21T05:56:50.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 0x7f17701a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1770112990 con 0x7f17701083d0 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1770112810 con 0x7f1770102ae0 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f177491b640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1770112b10 con 0x7f1770108d20 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38876/0 (socket says 172.21.15.28:38876) 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57668/0 (socket says 172.21.15.28:57668) 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176e59c640 1 -- 172.21.15.28:0/1178151313 learned_addr learned my addr 172.21.15.28:0/1178151313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.015+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 0x7f17701a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 -- 172.21.15.28:0/1178151313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 msgr2=0x7f17701a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 0x7f17701a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 -- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 msgr2=0x7f177019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 -- 172.21.15.28:0/1178151313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17701a2620 con 0x7f1770102ae0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176dd9b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f176e59c640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1760005940 tx=0x7f1760004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f175f7fe640 1 -- 172.21.15.28:0/1178151313 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176003f030 con 0x7f1770102ae0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f175f7fe640 1 -- 172.21.15.28:0/1178151313 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1760003aa0 con 0x7f1770102ae0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17701a28b0 con 0x7f1770102ae0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f175f7fe640 1 -- 172.21.15.28:0/1178151313 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760035070 con 0x7f1770102ae0 2024-09-21T05:56:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.016+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17701a9070 con 0x7f1770102ae0 2024-09-21T05:56:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 Processor -- start 2024-09-21T05:56:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 -- start start 2024-09-21T05:56:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f175f7fe640 1 -- 172.21.15.28:0/1178151313 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f1760042030 con 0x7f1770102ae0 2024-09-21T05:56:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 0x559ecbacd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 0x559ecbacd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb705390 0x559ecbacdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ecba43ea0 con 0x559ecb705390 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559ecba332d0 con 0x559ecb6fd1b0 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f177ae8f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559ecba370e0 con 0x559ecb726070 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f176e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 0x559ecbacd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f176dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb705390 0x559ecbacdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f176e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 0x559ecbacd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38890/0 (socket says 172.21.15.28:38890) 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 0x559ecbacd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.019+0000 7f176e59c640 1 -- 172.21.15.28:0/1023380940 learned_addr learned my addr 172.21.15.28:0/1023380940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 -- 172.21.15.28:0/1023380940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 msgr2=0x559ecbacd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 0x559ecbacd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 -- 172.21.15.28:0/1023380940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb705390 msgr2=0x559ecbacdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb705390 0x559ecbacdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 -- 172.21.15.28:0/1023380940 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ecb791650 con 0x559ecb726070 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 0x559ecbacd2c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1768009b80 tx=0x7f1768011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=181110a3b07315bf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f175cff9640 1 -- 172.21.15.28:0/1023380940 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768004050 con 0x559ecb726070 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f175cff9640 1 -- 172.21.15.28:0/1023380940 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17680054d0 con 0x559ecb726070 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.020+0000 7f175cff9640 1 -- 172.21.15.28:0/1023380940 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768004b80 con 0x559ecb726070 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f175f7fe640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 0x7f17580b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f176e59c640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 0x7f17580b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f175f7fe640 1 -- 172.21.15.28:0/1178151313 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f1760107dd0 con 0x7f1770102ae0 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 -- 172.21.15.28:0/1023380940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 msgr2=0x559ecbacd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 0x559ecbacd2c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1768009b80 tx=0x7f1768011810 comp rx=0 tx=0).stop 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f176e59c640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 0x7f17580b8c10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17601054f0 tx=0x7f176010b880 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 -- 172.21.15.28:0/1023380940 shutdown_connections 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb705390 0x559ecbacdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb6fd1b0 0x559ecbacd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1023380940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb726070 0x559ecbacd2c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 -- 172.21.15.28:0/1023380940 >> 172.21.15.28:0/1023380940 conn(0x559ecb71a460 msgr2=0x559ecbacb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 -- 172.21.15.28:0/1023380940 shutdown_connections 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.021+0000 7f177ae8f740 1 -- 172.21.15.28:0/1023380940 wait complete. 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 -- start start 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 Processor -- start 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 0x559ecbad8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb705390 0x559ecbad8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 0x559ecbae4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ecb894700 con 0x559ecb6fd1b0 2024-09-21T05:56:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559ecb9ce160 con 0x559ecb726070 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559ecbadbaa0 con 0x559ecb705390 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f177ae8f740 10 client.0 _ll_register_callbacks cb 0x559ecb9e79e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 0x559ecbad8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 0x559ecbad8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57688/0 (socket says 172.21.15.28:57688) 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.022+0000 7f176ed9d640 1 -- 172.21.15.28:0/3356250949 learned_addr learned my addr 172.21.15.28:0/3356250949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 0x559ecbae4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb705390 0x559ecbad8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 -- 172.21.15.28:0/3356250949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb705390 msgr2=0x559ecbad8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb705390 0x559ecbad8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 -- 172.21.15.28:0/3356250949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 msgr2=0x559ecbad8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 0x559ecbad8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 -- 172.21.15.28:0/3356250949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ecb9fbc90 con 0x559ecb726070 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 0x559ecbae4dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1768009710 tx=0x7f1768005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1942cca07c05a375 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f1755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768003d00 con 0x559ecb726070 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f1768028450 con 0x559ecb726070 2024-09-21T05:56:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.023+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17680054d0 con 0x559ecb726070 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1768028650 con 0x559ecb726070 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 10 client.27308 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x559ecbae77a0 con 0x559ecb726070 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17680054d0 con 0x559ecb726070 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f1768021030 con 0x559ecb726070 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 10 client.27308 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 10 client.27308 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 20 client.27308 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6e49f0 0x559ecbaeabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 4 client.27308 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ecba45df0 con 0x559ecb6e49f0 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.024+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6e49f0 0x559ecbaeabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.025+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6e49f0 0x559ecbaeabb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80a5da5d8eb14708 server_cookie=d84bab3ebe842b91 in_seq=0 out_seq=0 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.025+0000 7f1755ffb640 10 client.27308 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.025+0000 7f1755ffb640 1 -- 172.21.15.28:0/3356250949 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559ecba45df0 con 0x559ecb6e49f0 2024-09-21T05:56:50.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.025+0000 7f1755ffb640 10 client.27308 handle_command_reply: tid=0 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.025+0000 7f177ae8f740 1 client.27308 shutdown 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 msgr2=0x559ecbae4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 0x559ecbae4dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1768009710 tx=0x7f1768005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 shutdown_connections 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6e49f0 0x559ecbaeabb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb726070 0x559ecbae4dc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb705390 0x559ecbad8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f1755ffb640 0 client.27308 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3356250949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb6fd1b0 0x559ecbad8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 >> 172.21.15.28:0/3356250949 conn(0x559ecb71a460 msgr2=0x559ecbad0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 shutdown_connections 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 1 -- 172.21.15.28:0/3356250949 wait complete. 2024-09-21T05:56:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.026+0000 7f177ae8f740 20 client.27308 trim_cache size 0 max 16384 2024-09-21T05:56:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 Processor -- start 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 -- start start 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbacecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbada4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b1a50 con 0x559ecb726070 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b3590 con 0x559ecb705390 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f177ae8f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b2120 con 0x559ecb6fd1b0 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57704/0 (socket says 172.21.15.28:57704) 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.039+0000 7f176ed9d640 1 -- 172.21.15.28:0/1929650117 learned_addr learned my addr 172.21.15.28:0/1929650117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176dd9b640 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbada4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbacecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 -- 172.21.15.28:0/1929650117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 msgr2=0x559ecbacecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbacecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 -- 172.21.15.28:0/1929650117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 msgr2=0x559ecbada4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbada4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 -- 172.21.15.28:0/1929650117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ecbacf5b0 con 0x559ecb726070 2024-09-21T05:56:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f176ed9d640 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1768014040 tx=0x7f1768011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbeeed7996aa516 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f1757fff640 1 -- 172.21.15.28:0/1929650117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768003d00 con 0x559ecb726070 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f1757fff640 1 -- 172.21.15.28:0/1929650117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1768003ea0 con 0x559ecb726070 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.040+0000 7f1757fff640 1 -- 172.21.15.28:0/1929650117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768004e20 con 0x559ecb726070 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- 172.21.15.28:0/1929650117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 msgr2=0x559ecbaca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1768014040 tx=0x7f1768011810 comp rx=0 tx=0).stop 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- 172.21.15.28:0/1929650117 shutdown_connections 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbaca760 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbada4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 --2- 172.21.15.28:0/1929650117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbacecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- 172.21.15.28:0/1929650117 >> 172.21.15.28:0/1929650117 conn(0x559ecb71a460 msgr2=0x559ecbad9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- 172.21.15.28:0/1929650117 shutdown_connections 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- 172.21.15.28:0/1929650117 wait complete. 2024-09-21T05:56:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 -- start start 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.041+0000 7f177ae8f740 1 Processor -- start 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbadd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbadd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b2ec0 con 0x559ecb726070 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b27f0 con 0x559ecb705390 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17701afa50 con 0x559ecb6fd1b0 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f177ae8f740 10 client.0 _ll_register_callbacks cb 0x7f176010b750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:53092/0 (socket says 172.21.15.28:53092) 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 -- 172.21.15.28:0/3867605283 learned_addr learned my addr 172.21.15.28:0/3867605283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbadd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbadd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 -- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 msgr2=0x559ecbadd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbadd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 -- 172.21.15.28:0/3867605283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 msgr2=0x559ecbadd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbadd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.042+0000 7f176ed9d640 1 -- 172.21.15.28:0/3867605283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ecbb543b0 con 0x559ecb6fd1b0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f176ed9d640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f17680092b0 tx=0x7f1768005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dbb22575aab13f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768004580 con 0x559ecb6fd1b0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1768003ea0 con 0x559ecb6fd1b0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f1768004720 con 0x559ecb6fd1b0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1768004920 con 0x559ecb6fd1b0 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 10 client.27295 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x559ecbb55570 con 0x559ecb6fd1b0 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1768003ea0 con 0x559ecb6fd1b0 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f1768021070 con 0x559ecb6fd1b0 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 10 client.27295 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 10 client.27295 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 20 client.27295 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6b9da0 0x559ecbb57c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 4 client.27295 mds_command: new command op to 24680 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.043+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 --> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ecba418a0 con 0x559ecb6b9da0 2024-09-21T05:56:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6b9da0 0x559ecbb57c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f176dd9b640 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6b9da0 0x559ecbb57c40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae322304826c2cac server_cookie=40fd5b546f0f37 in_seq=0 out_seq=0 2024-09-21T05:56:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f1755ffb640 10 client.27295 ms_handle_connect on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f1755ffb640 1 -- 172.21.15.28:0/3867605283 <== mds.1 v2:172.21.15.129:6838/3631196486 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ecba418a0 con 0x559ecb6b9da0 2024-09-21T05:56:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f1755ffb640 10 client.27295 handle_command_reply: tid=0 2024-09-21T05:56:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 client.27295 shutdown 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 msgr2=0x559ecbadcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f17680092b0 tx=0x7f1768005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 shutdown_connections 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:6838/3631196486,v1:172.21.15.129:6839/3631196486] conn(0x559ecb6b9da0 0x559ecbb57c40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ecb726070 0x559ecbadd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f1755ffb640 0 client.27295 ms_handle_reset on v2:172.21.15.129:6838/3631196486 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559ecb705390 0x559ecbadd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 --2- 172.21.15.28:0/3867605283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559ecb6fd1b0 0x559ecbadcc40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.044+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 >> 172.21.15.28:0/3867605283 conn(0x559ecb71a460 msgr2=0x559ecbacea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 shutdown_connections 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177ae8f740 1 -- 172.21.15.28:0/3867605283 wait complete. 2024-09-21T05:56:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177ae8f740 20 client.27295 trim_cache size 0 max 16384 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 msgr2=0x7f17580b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 0x7f17580b8c10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17601054f0 tx=0x7f176010b880 comp rx=0 tx=0).stop 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 msgr2=0x7f177019c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1760005940 tx=0x7f1760004270 comp rx=0 tx=0).stop 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 shutdown_connections 2024-09-21T05:56:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1770108d20 0x7f17701a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f17580b6700 0x7f17580b8c10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17701083d0 0x7f177019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 --2- 172.21.15.28:0/1178151313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1770102ae0 0x7f177019c6b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 >> 172.21.15.28:0/1178151313 conn(0x7f1770079760 msgr2=0x7f177006bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 shutdown_connections 2024-09-21T05:56:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:50.045+0000 7f177491b640 1 -- 172.21.15.28:0/1178151313 wait complete. 2024-09-21T05:56:50.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:50 smithi028 ceph-mon[31227]: pgmap v624: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:50.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:50 smithi181 ceph-mon[34819]: pgmap v624: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:50.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:50 smithi129 ceph-mon[34885]: pgmap v624: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:50.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph tell mds.1:0 damage ls 2024-09-21T05:56:51.177 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 -- 172.21.15.28:0/3429806033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4102a60 msgr2=0x7f13c4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 --2- 172.21.15.28:0/3429806033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4102a60 0x7f13c4102e70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009900 tx=0x7f13b402a410 comp rx=0 tx=0).stop 2024-09-21T05:56:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 -- 172.21.15.28:0/3429806033 shutdown_connections 2024-09-21T05:56:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 --2- 172.21.15.28:0/3429806033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c4108cf0 0x7f13c410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 --2- 172.21.15.28:0/3429806033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c41083a0 0x7f13c41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 --2- 172.21.15.28:0/3429806033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4102a60 0x7f13c4102e70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 -- 172.21.15.28:0/3429806033 >> 172.21.15.28:0/3429806033 conn(0x7f13c4079730 msgr2=0x7f13c4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 -- 172.21.15.28:0/3429806033 shutdown_connections 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.646+0000 7f13ca390640 1 -- 172.21.15.28:0/3429806033 wait complete. 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 Processor -- start 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 -- start start 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c41083a0 0x7f13c419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13c4112960 con 0x7f13c4108cf0 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41127e0 con 0x7f13c4102a60 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.647+0000 7f13ca390640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f13c4112ae0 con 0x7f13c41083a0 2024-09-21T05:56:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57744/0 (socket says 172.21.15.28:57744) 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:38924/0 (socket says 172.21.15.28:38924) 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 -- 172.21.15.28:0/2357511488 learned_addr learned my addr 172.21.15.28:0/2357511488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c37fe640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c41083a0 0x7f13c419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 -- 172.21.15.28:0/2357511488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c41083a0 msgr2=0x7f13c419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c41083a0 0x7f13c419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 -- 172.21.15.28:0/2357511488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 msgr2=0x7f13c419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 -- 172.21.15.28:0/2357511488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c41a0390 con 0x7f13c4108cf0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c3fff640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c892d640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00eb10 tx=0x7f13bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c17fa640 1 -- 172.21.15.28:0/2357511488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc004b20 con 0x7f13c4108cf0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.648+0000 7f13c17fa640 1 -- 172.21.15.28:0/2357511488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13bc004cc0 con 0x7f13c4108cf0 2024-09-21T05:56:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.649+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c41a0680 con 0x7f13c4108cf0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.649+0000 7f13c17fa640 1 -- 172.21.15.28:0/2357511488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc010670 con 0x7f13c4108cf0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.649+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c4104130 con 0x7f13c4108cf0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.652+0000 7f13d0904740 1 Processor -- start 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.652+0000 7f13c17fa640 1 -- 172.21.15.28:0/2357511488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f13bc005020 con 0x7f13c4108cf0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.652+0000 7f13d0904740 1 -- start start 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fe6070 0x5633b438d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b438e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633b4303ea0 con 0x5633b3fc5390 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5633b42f32d0 con 0x5633b3fe6070 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13d0904740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5633b42f70e0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b438e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:53106/0 (socket says 172.21.15.28:53106) 2024-09-21T05:56:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 -- 172.21.15.28:0/384400762 learned_addr learned my addr 172.21.15.28:0/384400762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b438e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57752/0 (socket says 172.21.15.28:57752) 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c37fe640 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fe6070 0x5633b438d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 -- 172.21.15.28:0/384400762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fe6070 msgr2=0x5633b438d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fe6070 0x5633b438d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 -- 172.21.15.28:0/384400762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 msgr2=0x5633b438e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b438e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.653+0000 7f13c3fff640 1 -- 172.21.15.28:0/384400762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b4051650 con 0x5633b3fbd1b0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13c17fa640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 0x7f13ac0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13c3fff640 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009900 tx=0x7f13b4032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a7829b8a4182478 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13c37fe640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 0x7f13ac0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13c17fa640 1 -- 172.21.15.28:0/2357511488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f13bc0e7d60 con 0x7f13c4108cf0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13aa7fc640 1 -- 172.21.15.28:0/384400762 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4035050 con 0x5633b3fbd1b0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13aa7fc640 1 -- 172.21.15.28:0/384400762 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b40316a0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.654+0000 7f13aa7fc640 1 -- 172.21.15.28:0/384400762 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4004770 con 0x5633b3fbd1b0 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- 172.21.15.28:0/384400762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 msgr2=0x5633b438dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13c37fe640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 0x7f13ac0b8b90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fd0 tx=0x7f13b8009640 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009900 tx=0x7f13b4032f90 comp rx=0 tx=0).stop 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- 172.21.15.28:0/384400762 shutdown_connections 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b438e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b438dbe0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 --2- 172.21.15.28:0/384400762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fe6070 0x5633b438d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- 172.21.15.28:0/384400762 >> 172.21.15.28:0/384400762 conn(0x5633b3fda460 msgr2=0x5633b438b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- 172.21.15.28:0/384400762 shutdown_connections 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- 172.21.15.28:0/384400762 wait complete. 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.655+0000 7f13d0904740 1 -- start start 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 Processor -- start 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4398d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b4399240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fe6070 0x5633b43a5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633b4154700 con 0x5633b3fc5390 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5633b428e160 con 0x5633b3fbd1b0 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5633b439be20 con 0x5633b3fe6070 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b4399240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13d0904740 10 client.0 _ll_register_callbacks cb 0x5633b42a79e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b4399240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57766/0 (socket says 172.21.15.28:57766) 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4398d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13c37fe640 1 -- 172.21.15.28:0/2243157869 learned_addr learned my addr 172.21.15.28:0/2243157869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.656+0000 7f13c892d640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fe6070 0x5633b43a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 -- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fe6070 msgr2=0x5633b43a5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fe6070 0x5633b43a5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 -- 172.21.15.28:0/2243157869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 msgr2=0x5633b4399240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b4399240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 -- 172.21.15.28:0/2243157869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b42bbc90 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13c3fff640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4398d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f13b402a930 tx=0x7f13b40052d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=436f25a4de655ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4052060 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f13b40054f0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b40316a0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f13b4035020 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13d0904740 10 client.27326 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.657+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5633b43a7ad0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b40316a0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f13b4003710 con 0x5633b3fbd1b0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 10 client.27326 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 10 client.27326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 20 client.27326 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3fa49f0 0x5633b43aae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 4 client.27326 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633b4305df0 con 0x5633b3fa49f0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.658+0000 7f13c37fe640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3fa49f0 0x5633b43aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.659+0000 7f13a37fe640 10 client.27326 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.659+0000 7f13c37fe640 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3fa49f0 0x5633b43aae30 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=721bd06db7b44717 server_cookie=c4e0633c3ba7bdc7 in_seq=0 out_seq=0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13a37fe640 1 -- 172.21.15.28:0/2243157869 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5633b4305df0 con 0x5633b3fa49f0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13a37fe640 10 client.27326 handle_command_reply: tid=0 2024-09-21T05:56:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 client.27326 shutdown 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 msgr2=0x5633b4398d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4398d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f13b402a930 tx=0x7f13b40052d0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 shutdown_connections 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3fa49f0 0x5633b43aae30 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fe6070 0x5633b43a5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fc5390 0x5633b4399240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 --2- 172.21.15.28:0/2243157869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4398d00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13a37fe640 0 client.27326 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 >> 172.21.15.28:0/2243157869 conn(0x5633b3fda460 msgr2=0x5633b4390880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 shutdown_connections 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 1 -- 172.21.15.28:0/2243157869 wait complete. 2024-09-21T05:56:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.660+0000 7f13d0904740 20 client.27326 trim_cache size 0 max 16384 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 Processor -- start 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 -- start start 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4397bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fc5390 0x5633b439a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41ad5c0 con 0x5633b3fe6070 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41af100 con 0x5633b3fbd1b0 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13d0904740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41adc90 con 0x5633b3fc5390 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57776/0 (socket says 172.21.15.28:57776) 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13c892d640 1 -- 172.21.15.28:0/4198309115 learned_addr learned my addr 172.21.15.28:0/4198309115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.674+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4397bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c37fe640 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fc5390 0x5633b439a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 -- 172.21.15.28:0/4198309115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fc5390 msgr2=0x5633b439a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fc5390 0x5633b439a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 -- 172.21.15.28:0/4198309115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 msgr2=0x5633b438abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 -- 172.21.15.28:0/4198309115 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b438fbc0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c892d640 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13c3fff640 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4397bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009900 tx=0x7f13b402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b901fc777ed0690 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13a9ffb640 1 -- 172.21.15.28:0/4198309115 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4052070 con 0x5633b3fbd1b0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13a9ffb640 1 -- 172.21.15.28:0/4198309115 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b4003cb0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.675+0000 7f13a9ffb640 1 -- 172.21.15.28:0/4198309115 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b40049d0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 -- 172.21.15.28:0/4198309115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 msgr2=0x5633b4397bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4397bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009900 tx=0x7f13b402a410 comp rx=0 tx=0).stop 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 -- 172.21.15.28:0/4198309115 shutdown_connections 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b438abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fc5390 0x5633b439a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 --2- 172.21.15.28:0/4198309115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fbd1b0 0x5633b4397bf0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 -- 172.21.15.28:0/4198309115 >> 172.21.15.28:0/4198309115 conn(0x5633b3fda460 msgr2=0x5633b439a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 -- 172.21.15.28:0/4198309115 shutdown_connections 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.676+0000 7f13d0904740 1 -- 172.21.15.28:0/4198309115 wait complete. 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 -- start start 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 Processor -- start 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b43ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fc5390 0x5633b43ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b43abf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41aea30 con 0x5633b3fe6070 2024-09-21T05:56:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41ae360 con 0x5633b3fc5390 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41ab5c0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13d0904740 10 client.0 _ll_register_callbacks cb 0x7f13b8009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b43abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b43ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13c892d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b43abf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57790/0 (socket says 172.21.15.28:57790) 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13c892d640 1 -- 172.21.15.28:0/959186327 learned_addr learned my addr 172.21.15.28:0/959186327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.677+0000 7f13c37fe640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fc5390 0x5633b43ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 -- 172.21.15.28:0/959186327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fc5390 msgr2=0x5633b43ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fc5390 0x5633b43ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 -- 172.21.15.28:0/959186327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 msgr2=0x5633b43abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b43abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 -- 172.21.15.28:0/959186327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b43ac5a0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13c3fff640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b43ae520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f13b40092a0 tx=0x7f13b4005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be77f48870d42bb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-09-21T05:56:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4052070 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f13b40035f0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.678+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b4003cb0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f13b4003e60 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 10 client.27313 fetch_fsmap learned FSMap version 54 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5633b44153e0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b4003cb0 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 54) ==== 8362+0+0 (secure 0 0 0) 0x7f13b4004c90 con 0x5633b3fbd1b0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 10 client.27313 fetch_fsmap finished waiting for FSMap version 54 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 10 client.27313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 20 client.27313 populate_metadata read hostname 'smithi028' 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3f79da0 0x5633b4417950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 4 client.27313 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.679+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 --> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633b43018a0 con 0x5633b3f79da0 2024-09-21T05:56:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13c37fe640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3f79da0 0x5633b4417950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13c37fe640 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3f79da0 0x5633b4417950 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=888e67ddd852b624 server_cookie=4d1d9b3170fa8682 in_seq=0 out_seq=0 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13a37fe640 10 client.27313 ms_handle_connect on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13a37fe640 1 -- 172.21.15.28:0/959186327 <== mds.0 v2:172.21.15.181:6838/3430071220 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633b43018a0 con 0x5633b3f79da0 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13a37fe640 10 client.27313 handle_command_reply: tid=0 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.680+0000 7f13d0904740 1 client.27313 shutdown 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 msgr2=0x5633b43ae520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b43ae520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f13b40092a0 tx=0x7f13b4005a20 comp rx=0 tx=0).stop 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 shutdown_connections 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:6838/3430071220,v1:172.21.15.181:6839/3430071220] conn(0x5633b3f79da0 0x5633b4417950 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633b3fe6070 0x5633b43abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633b3fc5390 0x5633b43ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 --2- 172.21.15.28:0/959186327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5633b3fbd1b0 0x5633b43ae520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 >> 172.21.15.28:0/959186327 conn(0x5633b3fda460 msgr2=0x5633b439a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13a37fe640 0 client.27313 ms_handle_reset on v2:172.21.15.181:6838/3430071220 2024-09-21T05:56:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 shutdown_connections 2024-09-21T05:56:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 1 -- 172.21.15.28:0/959186327 wait complete. 2024-09-21T05:56:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.681+0000 7f13d0904740 20 client.27313 trim_cache size 0 max 16384 2024-09-21T05:56:51.691 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 msgr2=0x7f13ac0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 0x7f13ac0b8b90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fd0 tx=0x7f13b8009640 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 msgr2=0x7f13c419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00eb10 tx=0x7f13bc00c2d0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 shutdown_connections 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f13ac0b6680 0x7f13ac0b8b90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13c4108cf0 0x7f13c419fc60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13c41083a0 0x7f13c419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 --2- 172.21.15.28:0/2357511488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13c4102a60 0x7f13c419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 >> 172.21.15.28:0/2357511488 conn(0x7f13c4079730 msgr2=0x7f13c4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 shutdown_connections 2024-09-21T05:56:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:51.682+0000 7f13ca390640 1 -- 172.21.15.28:0/2357511488 wait complete. 2024-09-21T05:56:52.486 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks/thrasher.py", line 50, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-21T05:56:52.487 INFO:tasks.fwd_scrub:done joining 2024-09-21T05:56:52.487 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-21T05:56:52.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T05:56:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 -- 172.21.15.28:0/3134483327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4508108c00 msgr2=0x7f450810f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 --2- 172.21.15.28:0/3134483327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4508108c00 0x7f450810f430 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4500009ca0 tx=0x7f450002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 -- 172.21.15.28:0/3134483327 shutdown_connections 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 --2- 172.21.15.28:0/3134483327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4508108c00 0x7f450810f430 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 --2- 172.21.15.28:0/3134483327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45081082b0 0x7f45081086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 --2- 172.21.15.28:0/3134483327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508102980 0x7f4508102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 -- 172.21.15.28:0/3134483327 >> 172.21.15.28:0/3134483327 conn(0x7f45080fe380 msgr2=0x7f45081007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 -- 172.21.15.28:0/3134483327 shutdown_connections 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 -- 172.21.15.28:0/3134483327 wait complete. 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.628+0000 7f450ff13640 1 Processor -- start 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 -- start start 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4508102980 0x7f450819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508108c00 0x7f450819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45081128c0 con 0x7f45081082b0 2024-09-21T05:56:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4508112740 con 0x7f4508102980 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450ff13640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4508112a40 con 0x7f4508108c00 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57812/0 (socket says 172.21.15.28:57812) 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 -- 172.21.15.28:0/1241583027 learned_addr learned my addr 172.21.15.28:0/1241583027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450dcaf640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4508102980 0x7f450819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450e4b0640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508108c00 0x7f450819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 -- 172.21.15.28:0/1241583027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508108c00 msgr2=0x7f450819bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508108c00 0x7f450819bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 -- 172.21.15.28:0/1241583027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4508102980 msgr2=0x7f450819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4508102980 0x7f450819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.629+0000 7f450d4ae640 1 -- 172.21.15.28:0/1241583027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45081a6470 con 0x7f45081082b0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f450d4ae640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9f0 tx=0x7f44f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f800ccc0 con 0x7f45081082b0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f80108e0 con 0x7f45081082b0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8018860 con 0x7f45081082b0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45081a6760 con 0x7f45081082b0 2024-09-21T05:56:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.630+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45081a6cb0 con 0x7f45081082b0 2024-09-21T05:56:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.631+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4508103bf0 con 0x7f45081082b0 2024-09-21T05:56:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.634+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f44f8010430 con 0x7f45081082b0 2024-09-21T05:56:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.635+0000 7f44feffd640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 0x7f44f40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.635+0000 7f450dcaf640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 0x7f44f40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.635+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f44f80e7eb0 con 0x7f45081082b0 2024-09-21T05:56:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.635+0000 7f450dcaf640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 0x7f44f40b8b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f450806b2a0 tx=0x7f4504005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.637+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f44f80b27b0 con 0x7f45081082b0 2024-09-21T05:56:52.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:52 smithi181 ceph-mon[34819]: pgmap v625: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:52.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:52 smithi028 ceph-mon[31227]: pgmap v625: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:52.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.847+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f450810aa50 con 0x7f45081082b0 2024-09-21T05:56:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.849+0000 7f44feffd640 1 -- 172.21.15.28:0/1241583027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+23279 (secure 0 0 0) 0x7f44f80b76f0 con 0x7f45081082b0 2024-09-21T05:56:52.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:56:52.851 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":78,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:56:06.343760+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":78,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 msgr2=0x7f44f40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 0x7f44f40b8b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f450806b2a0 tx=0x7f4504005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 msgr2=0x7f450819b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9f0 tx=0x7f44f800eeb0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 shutdown_connections 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4508108c00 0x7f450819bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f44f40b6680 0x7f44f40b8b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45081082b0 0x7f450819b6f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 --2- 172.21.15.28:0/1241583027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4508102980 0x7f450819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 >> 172.21.15.28:0/1241583027 conn(0x7f45080fe380 msgr2=0x7f4508105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 shutdown_connections 2024-09-21T05:56:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.851+0000 7f450ff13640 1 -- 172.21.15.28:0/1241583027 wait complete. 2024-09-21T05:56:52.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:52 smithi129 ceph-mon[34885]: pgmap v625: 97 pgs: 12 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 59 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:52.865 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T05:40:11.086256+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-21T05:44:35.312867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-21T05:44:37.712198+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 78, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T05:56:52.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- 172.21.15.28:0/1871423789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c1047c0 msgr2=0x7f412c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 --2- 172.21.15.28:0/1871423789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c1047c0 0x7f412c104bd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f411c009900 tx=0x7f411c02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- 172.21.15.28:0/1871423789 shutdown_connections 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 --2- 172.21.15.28:0/1871423789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c10dd70 0x7f412c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 --2- 172.21.15.28:0/1871423789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c10d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 --2- 172.21.15.28:0/1871423789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c1047c0 0x7f412c104bd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- 172.21.15.28:0/1871423789 >> 172.21.15.28:0/1871423789 conn(0x7f412c0fe3b0 msgr2=0x7f412c100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- 172.21.15.28:0/1871423789 shutdown_connections 2024-09-21T05:56:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- 172.21.15.28:0/1871423789 wait complete. 2024-09-21T05:56:52.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 Processor -- start 2024-09-21T05:56:52.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.987+0000 7f41329f1640 1 -- start start 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 0x7f412c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f412c1139d0 con 0x7f412c10dd70 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f412c113850 con 0x7f412c1047c0 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f41329f1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f412c113b50 con 0x7f412c1051a0 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f412bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 0x7f412c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f412b7fe640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57822/0 (socket says 172.21.15.28:57822) 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 -- 172.21.15.28:0/1014300274 learned_addr learned my addr 172.21.15.28:0/1014300274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 -- 172.21.15.28:0/1014300274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 msgr2=0x7f412c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.988+0000 7f4130f8e640 1 -- 172.21.15.28:0/1014300274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 msgr2=0x7f412c19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f4130f8e640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 0x7f412c19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f4130f8e640 1 -- 172.21.15.28:0/1014300274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c1a0380 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f412bfff640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 0x7f412c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f412b7fe640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f4130f8e640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f412400eaa0 tx=0x7f412400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412400cdb0 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4124005020 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c1a0670 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41240116d0 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.989+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c1a6e00 con 0x7f412c10dd70 2024-09-21T05:56:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.990+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c104bd0 con 0x7f412c10dd70 2024-09-21T05:56:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.993+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f41240051d0 con 0x7f412c10dd70 2024-09-21T05:56:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.994+0000 7f41297fa640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 0x7f41140b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.994+0000 7f412bfff640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 0x7f41140b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.994+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f41240e78a0 con 0x7f412c10dd70 2024-09-21T05:56:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.994+0000 7f412bfff640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 0x7f41140b8b90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f411c031440 tx=0x7f411c032ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:52.999+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4124016030 con 0x7f412c10dd70 2024-09-21T05:56:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.199+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f412c1a70a0 con 0x7f412c10dd70 2024-09-21T05:56:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.199+0000 7f41297fa640 1 -- 172.21.15.28:0/1014300274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f41240b21a0 con 0x7f412c10dd70 2024-09-21T05:56:53.202 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 msgr2=0x7f41140b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 0x7f41140b8b90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f411c031440 tx=0x7f411c032ed0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 msgr2=0x7f412c19fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f412400eaa0 tx=0x7f412400ef60 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 shutdown_connections 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f41140b6680 0x7f41140b8b90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f412c10dd70 0x7f412c19fc80 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f412c1051a0 0x7f412c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 --2- 172.21.15.28:0/1014300274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f412c1047c0 0x7f412c19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 >> 172.21.15.28:0/1014300274 conn(0x7f412c0fe3b0 msgr2=0x7f412c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:53.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 shutdown_connections 2024-09-21T05:56:53.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.202+0000 7f41329f1640 1 -- 172.21.15.28:0/1014300274 wait complete. 2024-09-21T05:56:53.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-21T05:56:53.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 -- 172.21.15.28:0/1529319239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 msgr2=0x7f91a810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1529319239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a810d880 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f919c009900 tx=0x7f919c02a410 comp rx=0 tx=0).stop 2024-09-21T05:56:53.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 -- 172.21.15.28:0/1529319239 shutdown_connections 2024-09-21T05:56:53.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1529319239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 0x7f91a8110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1529319239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a810d880 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1529319239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a8104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 -- 172.21.15.28:0/1529319239 >> 172.21.15.28:0/1529319239 conn(0x7f91a8079630 msgr2=0x7f91a8079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:53.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 -- 172.21.15.28:0/1529319239 shutdown_connections 2024-09-21T05:56:53.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.340+0000 7f91ae28c640 1 -- 172.21.15.28:0/1529319239 wait complete. 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 Processor -- start 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 -- start start 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 0x7f91a819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81139d0 con 0x7f91a8105240 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8113850 con 0x7f91a8104810 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ae28c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8113b50 con 0x7f91a810ddc0 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57850/0 (socket says 172.21.15.28:57850) 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91a7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91a77fe640 1 -- 172.21.15.28:0/1648418764 learned_addr learned my addr 172.21.15.28:0/1648418764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.341+0000 7f91ac829640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 0x7f91a819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 -- 172.21.15.28:0/1648418764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 msgr2=0x7f91a819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 0x7f91a819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 -- 172.21.15.28:0/1648418764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 msgr2=0x7f91a819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 -- 172.21.15.28:0/1648418764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a81a02e0 con 0x7f91a8105240 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a7fff640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a77fe640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f919c032fd0 tx=0x7f919c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c03f030 con 0x7f91a8105240 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a81a0570 con 0x7f91a8105240 2024-09-21T05:56:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f919c004490 con 0x7f91a8105240 2024-09-21T05:56:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a81a6d60 con 0x7f91a8105240 2024-09-21T05:56:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.342+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c049060 con 0x7f91a8105240 2024-09-21T05:56:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.343+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a8104c20 con 0x7f91a8105240 2024-09-21T05:56:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.345+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f919c042030 con 0x7f91a8105240 2024-09-21T05:56:53.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.346+0000 7f91a57fa640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 0x7f91900b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.346+0000 7f91a7fff640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 0x7f91900b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.346+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f919c107e60 con 0x7f91a8105240 2024-09-21T05:56:53.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.346+0000 7f91a7fff640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 0x7f91900b8b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9198009fc0 tx=0x7f9198005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.349+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f919c0d2700 con 0x7f91a8105240 2024-09-21T05:56:53.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.553+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f91a8108f30 con 0x7f91a8105240 2024-09-21T05:56:53.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.553+0000 7f91a57fa640 1 -- 172.21.15.28:0/1648418764 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f919c0d7640 con 0x7f91a8105240 2024-09-21T05:56:53.555 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 msgr2=0x7f91900b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 0x7f91900b8b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9198009fc0 tx=0x7f9198005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 msgr2=0x7f91a819a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f919c032fd0 tx=0x7f919c031a20 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 shutdown_connections 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91a810ddc0 0x7f91a819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f91900b6680 0x7f91900b8b90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a8105240 0x7f91a819a950 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 --2- 172.21.15.28:0/1648418764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91a8104810 0x7f91a819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 >> 172.21.15.28:0/1648418764 conn(0x7f91a8079630 msgr2=0x7f91a80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 shutdown_connections 2024-09-21T05:56:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.556+0000 7f91ae28c640 1 -- 172.21.15.28:0/1648418764 wait complete. 2024-09-21T05:56:53.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-21T05:56:53.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 -- 172.21.15.28:0/2108878635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678108c10 msgr2=0x7f767810f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 --2- 172.21.15.28:0/2108878635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678108c10 0x7f767810f440 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7670009ca0 tx=0x7f767002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 -- 172.21.15.28:0/2108878635 shutdown_connections 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 --2- 172.21.15.28:0/2108878635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678108c10 0x7f767810f440 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 --2- 172.21.15.28:0/2108878635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76781082c0 0x7f76781086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 --2- 172.21.15.28:0/2108878635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7678102990 0x7f7678102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 -- 172.21.15.28:0/2108878635 >> 172.21.15.28:0/2108878635 conn(0x7f76780fe3b0 msgr2=0x7f7678100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 -- 172.21.15.28:0/2108878635 shutdown_connections 2024-09-21T05:56:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.692+0000 7f767cdc3640 1 -- 172.21.15.28:0/2108878635 wait complete. 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 Processor -- start 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 -- start start 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76781082c0 0x7f767819b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7678108c10 0x7f767819bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76781128d0 con 0x7f7678102990 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7678112750 con 0x7f7678108c10 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767cdc3640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7678112a50 con 0x7f76781082c0 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57878/0 (socket says 172.21.15.28:57878) 2024-09-21T05:56:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.693+0000 7f767659c640 1 -- 172.21.15.28:0/1961390575 learned_addr learned my addr 172.21.15.28:0/1961390575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f7675d9b640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76781082c0 0x7f767819b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f7676d9d640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7678108c10 0x7f767819bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 -- 172.21.15.28:0/1961390575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76781082c0 msgr2=0x7f767819b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76781082c0 0x7f767819b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 -- 172.21.15.28:0/1961390575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7678108c10 msgr2=0x7f767819bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7678108c10 0x7f767819bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 -- 172.21.15.28:0/1961390575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76781a6430 con 0x7f7678102990 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767659c640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f766800e9f0 tx=0x7f766800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766800cd80 con 0x7f7678102990 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76781a6720 con 0x7f7678102990 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76680108e0 con 0x7f7678102990 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76781a6cc0 con 0x7f7678102990 2024-09-21T05:56:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.694+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668018920 con 0x7f7678102990 2024-09-21T05:56:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.695+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7678102da0 con 0x7f7678102990 2024-09-21T05:56:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.697+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f7668010430 con 0x7f7678102990 2024-09-21T05:56:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.698+0000 7f76677fe640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 0x7f76600b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.698+0000 7f7675d9b640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 0x7f76600b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.698+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f76680e7b10 con 0x7f7678102990 2024-09-21T05:56:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.698+0000 7f7675d9b640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 0x7f76600b8b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7678105b90 tx=0x7f766c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.702+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f76680adfd0 con 0x7f7678102990 2024-09-21T05:56:53.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:53 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1241583027' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:56:53.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:53 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1014300274' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:56:53.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:53 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1241583027' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:56:53.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:53 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1014300274' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:56:53.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:53 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1241583027' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:56:53.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:53 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1014300274' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T05:56:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.904+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f767802e640 con 0x7f7678102990 2024-09-21T05:56:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.904+0000 7f76677fe640 1 -- 172.21.15.28:0/1961390575 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f76680b23b0 con 0x7f7678102990 2024-09-21T05:56:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-21T05:56:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 msgr2=0x7f76600b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 0x7f76600b8b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7678105b90 tx=0x7f766c005f90 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 msgr2=0x7f767819b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f766800e9f0 tx=0x7f766800eeb0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 shutdown_connections 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7678108c10 0x7f767819bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76781082c0 0x7f767819b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f76600b6680 0x7f76600b8b90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 --2- 172.21.15.28:0/1961390575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7678102990 0x7f767819b120 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 >> 172.21.15.28:0/1961390575 conn(0x7f76780fe3b0 msgr2=0x7f76781007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.906+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 shutdown_connections 2024-09-21T05:56:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:53.907+0000 7f767cdc3640 1 -- 172.21.15.28:0/1961390575 wait complete. 2024-09-21T05:56:53.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-21T05:56:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 -- 172.21.15.28:0/4221438379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109d60 msgr2=0x7f83c8110590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/4221438379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109d60 0x7f83c8110590 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009900 tx=0x7f83c002a410 comp rx=0 tx=0).stop 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 -- 172.21.15.28:0/4221438379 shutdown_connections 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/4221438379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109d60 0x7f83c8110590 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/4221438379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109410 0x7f83c8109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/4221438379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c8103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 -- 172.21.15.28:0/4221438379 >> 172.21.15.28:0/4221438379 conn(0x7f83c80ff500 msgr2=0x7f83c8101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 -- 172.21.15.28:0/4221438379 shutdown_connections 2024-09-21T05:56:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.039+0000 7f83cfdf1640 1 -- 172.21.15.28:0/4221438379 wait complete. 2024-09-21T05:56:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 Processor -- start 2024-09-21T05:56:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 -- start start 2024-09-21T05:56:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 0x7f83c819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83c8113a20 con 0x7f83c8109410 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f83c81138a0 con 0x7f83c8109d60 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cfdf1640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f83c8113ba0 con 0x7f83c8103ae0 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cd38c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cd38c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57908/0 (socket says 172.21.15.28:57908) 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cd38c640 1 -- 172.21.15.28:0/336497708 learned_addr learned my addr 172.21.15.28:0/336497708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.040+0000 7f83cdb8d640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83ce38e640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 0x7f83c819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 -- 172.21.15.28:0/336497708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 msgr2=0x7f83c819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 -- 172.21.15.28:0/336497708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 msgr2=0x7f83c819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 0x7f83c819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 -- 172.21.15.28:0/336497708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c81a02f0 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cdb8d640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83ce38e640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 0x7f83c819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cd38c640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9f0 tx=0x7f83bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc00ccc0 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83bc0108e0 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c81a05e0 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc018860 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.041+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c81a6d70 con 0x7f83c8109410 2024-09-21T05:56:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.042+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c8103ef0 con 0x7f83c8109410 2024-09-21T05:56:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.044+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f83bc010430 con 0x7f83c8109410 2024-09-21T05:56:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.045+0000 7f83c6ffd640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 0x7f83b00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:56:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.045+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8572+0+0 (secure 0 0 0) 0x7f83bc0e7fc0 con 0x7f83c8109410 2024-09-21T05:56:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.045+0000 7f83cdb8d640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 0x7f83b00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:56:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.045+0000 7f83cdb8d640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 0x7f83b00b8b90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005fd0 tx=0x7f83b8009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:56:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.048+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f83bc0ae480 con 0x7f83c8109410 2024-09-21T05:56:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.309+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f83c802e640 con 0x7f83c8109410 2024-09-21T05:56:54.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.317+0000 7f83c6ffd640 1 -- 172.21.15.28:0/336497708 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+834052 (secure 0 0 0) 0x7f83bc0b2860 con 0x7f83c8109410 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 25, 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 14223, 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4211147350 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4211147350 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/4211147350", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-09-21T05:38:12.622330+0000", 2024-09-21T05:56:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540719139924082687, 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24148, 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540719139924082687, 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T05:56:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T05:56:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T05:56:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T05:56:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T05:56:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T05:56:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-21T05:56:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-21T05:56:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-21T05:56:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-21T05:56:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-21T05:56:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-21T05:56:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-21T05:56:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T05:56:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nginx": { 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nginx", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nginx container image", 2024-09-21T05:56:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_oauth2_proxy": { 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "oauth2-proxy container image", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba_metrics": { 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba_metrics", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T05:56:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-21T05:56:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T05:56:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T05:56:54.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T05:56:54.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T05:56:54.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-21T05:56:54.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-21T05:56:54.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T05:56:54.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.419 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-21T05:56:54.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T05:56:54.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_count_max": { 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_interval": { 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_interval", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T05:56:54.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.440 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-09-21T05:56:54.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T05:56:54.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T05:56:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T05:56:54.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T05:56:54.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.476 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.477 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.478 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.479 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T05:56:54.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.493 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.494 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.498 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.505 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:54.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "sso_oauth2": { 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sso_oauth2", 2024-09-21T05:56:54.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.513 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.529 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-21T05:56:54.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-21T05:56:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-21T05:56:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.541 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.543 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.546 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.547 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.548 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:54.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-21T05:56:54.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-21T05:56:54.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.555 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.566 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.567 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.570 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T05:56:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-21T05:56:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-21T05:56:54.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max_requests": { 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_requests", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-21T05:56:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-21T05:56:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-21T05:56:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-21T05:56:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-21T05:56:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-21T05:56:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-21T05:56:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-21T05:56:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-21T05:56:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-21T05:56:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-21T05:56:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-21T05:56:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smb", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "internal_store_backend": { 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "internal_store_backend", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "update_orchestration": { 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "update_orchestration", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-21T05:56:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T05:56:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T05:56:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-21T05:56:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-21T05:56:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-21T05:56:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T05:56:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24164, 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540719139924082687, 2024-09-21T05:56:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-21T05:56:54.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.827 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.832 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-21T05:56:54.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.838 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:54.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T05:56:54.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T05:56:54.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T05:56:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T05:56:54.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-21T05:56:54.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.855 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.856 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-21T05:56:54.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.858 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.859 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nginx": { 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nginx", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nginx container image", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-21T05:56:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_oauth2_proxy": { 2024-09-21T05:56:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "oauth2-proxy container image", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-21T05:56:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-21T05:56:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba_metrics": { 2024-09-21T05:56:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba_metrics", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-21T05:56:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-21T05:56:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-21T05:56:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-21T05:56:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-21T05:56:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-21T05:56:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-21T05:56:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T05:56:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-21T05:56:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_count_max": { 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T05:56:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_interval": { 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_interval", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T05:56:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T05:56:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T05:56:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-21T05:56:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T05:56:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T05:56:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T05:56:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T05:56:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T05:56:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T05:56:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-21T05:56:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-21T05:56:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-21T05:56:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-21T05:56:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-21T05:56:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-21T05:56:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-21T05:56:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "sso_oauth2": { 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sso_oauth2", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-21T05:56:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:54.995 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:54.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:54.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.006 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.008 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-21T05:56:55.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-21T05:56:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-21T05:56:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-21T05:56:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-21T05:56:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:55.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-21T05:56:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:55.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.155 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max_requests": { 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_requests", 2024-09-21T05:56:55.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-21T05:56:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.164 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.174 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-21T05:56:55.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.178 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.192 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.193 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.195 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smb", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "internal_store_backend": { 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "internal_store_backend", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "update_orchestration": { 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "update_orchestration", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.209 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.216 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.220 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.224 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.228 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-21T05:56:55.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T05:56:55.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.259 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-21T05:56:55.262 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.263 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.268 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.269 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.270 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-21T05:56:55.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-09-21T05:56:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.290 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-21T05:56:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nginx": { 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nginx", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nginx container image", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-21T05:56:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_oauth2_proxy": { 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "oauth2-proxy container image", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba_metrics": { 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba_metrics", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-21T05:56:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-21T05:56:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-21T05:56:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-21T05:56:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T05:56:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_count_max": { 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_keepalive_interval": { 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_keepalive_interval", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-21T05:56:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T05:56:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T05:56:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T05:56:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T05:56:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T05:56:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T05:56:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-21T05:56:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.453 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "sso_oauth2": { 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sso_oauth2", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-21T05:56:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:55.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-21T05:56:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.470 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.480 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.481 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-21T05:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-21T05:56:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T05:56:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.504 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.507 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.509 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.512 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.517 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-21T05:56:55.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T05:56:55.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.521 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-21T05:56:55.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.523 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.525 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-21T05:56:55.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.530 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.540 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.544 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.547 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-21T05:56:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-21T05:56:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-21T05:56:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-21T05:56:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-21T05:56:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max_requests": { 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_requests", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smb", 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "internal_store_backend": { 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "internal_store_backend", 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.674 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "update_orchestration": { 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "update_orchestration", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.679 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.687 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.702 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-21T05:56:55.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-21T05:56:55.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.710 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.712 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-21T05:56:55.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-21T05:56:55.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-21T05:56:55.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-21T05:56:55.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-21T05:56:55.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.719 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.720 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.726 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.732 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.733 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.736 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.742 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-21T05:56:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-21T05:56:55.749 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.750 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-21T05:56:55.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-21T05:56:55.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.753 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.757 INFO:teuthology.orchestra.run.smithi028.stdout: "sqlite3_killpoint": { 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sqlite3_killpoint", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "dev", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-21T05:56:55.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-21T05:56:55.762 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-21T05:56:55.763 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-21T05:56:55.764 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-09-21T05:56:55.765 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 5, 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3789081143 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2980400559 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 728118779 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3021872015 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 243174886 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3401390844 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.770 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-21T05:56:55.771 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 msgr2=0x7f83b00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 0x7f83b00b8b90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005fd0 tx=0x7f83b8009500 comp rx=0 tx=0).stop 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 msgr2=0x7f83c819a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9f0 tx=0x7f83bc00eeb0 comp rx=0 tx=0).stop 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 shutdown_connections 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83c8109d60 0x7f83c819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f83b00b6680 0x7f83b00b8b90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8109410 0x7f83c819a960 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 --2- 172.21.15.28:0/336497708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83c8103ae0 0x7f83c819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:56:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.325+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 >> 172.21.15.28:0/336497708 conn(0x7f83c80ff500 msgr2=0x7f83c81069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:56:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.326+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 shutdown_connections 2024-09-21T05:56:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:56:54.326+0000 7f83cfdf1640 1 -- 172.21.15.28:0/336497708 wait complete. 2024-09-21T05:56:55.814 DEBUG:tasks.check_counter:Getting stats from a 2024-09-21T05:56:55.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.a perf dump 2024-09-21T05:56:55.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:54 smithi028 ceph-mon[31227]: pgmap v626: 97 pgs: 10 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 65 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:55.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:54 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1648418764' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:54 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1961390575' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:54 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:56:55.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:54 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/336497708' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:54 smithi129 ceph-mon[34885]: pgmap v626: 97 pgs: 10 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 65 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:54 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1648418764' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:54 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1961390575' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:54 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:54 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/336497708' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:54 smithi181 ceph-mon[34819]: pgmap v626: 97 pgs: 10 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 65 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:54 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1648418764' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:54 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1961390575' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T05:56:55.816 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:54 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T05:56:55.817 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:54 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/336497708' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T05:56:56.089 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1411, 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1413, 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2002281, 2024-09-21T05:56:56.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448699, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.161196404, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.063794764, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.038757012, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.047149532, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1411, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.074020123, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000052459 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:56.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 571, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 723, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 141032, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 537040, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-21T05:56:56.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.076537961, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.034550062, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.014105976, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.021645209, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 721, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.036539140, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000050678 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:56:56.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 355712 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 252, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 217, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1046412, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 126894, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.041145192, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.017770033, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.032506149, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000026649, 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:56.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 217, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.019821076, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000091341 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1046144, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 126576 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-21T05:56:56.600 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3303, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.098682840, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000029876 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-21T05:56:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-21T05:56:56.602 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 19, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 19, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 19, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-09-21T05:56:56.603 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-21T05:56:56.604 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-09-21T05:56:56.605 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-21T05:56:56.606 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:56:56.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-09-21T05:56:56.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 1171, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 3, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 5780756, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 5778151, 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-21T05:56:56.609 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 1171 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 4, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 127, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 123, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 2, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 14, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 12, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 19, 2024-09-21T05:56:56.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 142, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 123, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 119896, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 318582784 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-21T05:56:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-21T05:56:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-21T05:56:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-21T05:56:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-21T05:56:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1699, 2024-09-21T05:56:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:56:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-21T05:56:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 189383, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 50, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-21T05:56:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8376, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 185, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 26928, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 108, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-21T05:56:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 1982, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 1982, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1982, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 77.289895073, 2024-09-21T05:56:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.038995910 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1982, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1983 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 1982, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 1982, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 1215, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 765, 2024-09-21T05:56:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-21T05:56:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-21T05:56:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 78, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 7, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 5, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 5, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-21T05:56:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T05:56:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2230, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2973493, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2230, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2230, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2973493, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1982, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1586457, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1982, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1586457, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-21T05:56:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1982, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1982, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1982, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1982, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-21T05:56:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55c4dac95380": { 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:56.640 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:56.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: pgmap v627: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:56:56.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:56 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: pgmap v627: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:56:56.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:56:56.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:56 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: pgmap v627: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:56.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T05:56:56.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T05:56:56.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:56 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' 2024-09-21T05:56:57.260 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-21T05:56:57.260 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-21T05:56:57.260 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-09-21T05:56:57.260 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:56:57.260 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:56:57.261 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-09-21T05:56:57.261 DEBUG:tasks.check_counter:Getting stats from d 2024-09-21T05:56:57.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.d perf dump 2024-09-21T05:56:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 217, 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 210, 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 838576, 2024-09-21T05:56:58.035 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 127872, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.042181515, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.016993557, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.032466272, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 210, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017274038, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000082257 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:58.036 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 838576, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 127872 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 157, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 160, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 107824, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.014185381, 2024-09-21T05:56:58.037 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.011153477, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001427311, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 155, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.008011903, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000051689 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:56:58.038 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 107824 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.000468362, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000382319, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:56:58.039 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-09-21T05:56:58.040 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:56:58.041 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-21T05:56:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149047, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-21T05:56:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 968, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-21T05:56:58.044 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 214, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 805526, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 214, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:58.045 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 214, 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 805526, 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:58.046 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:56:58.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:56:58 smithi028 ceph-mon[31227]: pgmap v628: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-21T05:56:58.744 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-09-21T05:56:58.745 DEBUG:tasks.check_counter:Getting stats from g 2024-09-21T05:56:58.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.g perf dump 2024-09-21T05:56:58.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:56:58 smithi181 ceph-mon[34819]: pgmap v628: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:56:58 smithi129 ceph-mon[34885]: pgmap v628: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-21T05:56:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 642, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 642, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 217257, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 203333, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.088502567, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.041168490, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.019906585, 2024-09-21T05:56:59.533 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.020428394, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 641, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.039767019, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000062039 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T05:56:59.534 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1501, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1468, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2843584, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 525279, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.150246612, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.057644558, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.057309727, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.037235644, 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:59.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1467, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.068985410, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000047024 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1043472, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 126768 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:56:59.536 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 157, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1508, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 356869, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.013857650, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.010350747, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002129630, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000062205, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 155, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.007498521, 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000048377 2024-09-21T05:56:59.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 355904 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-21T05:56:59.538 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3233, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.088972241, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000027520 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-21T05:56:59.540 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 29, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 29, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 29, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-09-21T05:56:59.541 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-21T05:56:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-21T05:56:59.543 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-21T05:56:59.544 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:56:59.545 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_started": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_succeeded": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "uninline_write_failed": 0 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-09-21T05:56:59.546 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 1063, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 3, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-09-21T05:56:59.547 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 5675922, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 5673317, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 1063 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 4, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 153, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 149, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 2, 2024-09-21T05:56:59.548 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 15, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 13, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 29, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 178, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 149, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 117880, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 317468672 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-21T05:56:59.549 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.552 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-21T05:56:59.553 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.554 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.556 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-21T05:56:59.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.558 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-09-21T05:56:59.561 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1706, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:56:59.562 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:56:59.563 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-21T05:56:59.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 197463, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 54, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:56:59.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8376, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 185, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 33728, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 148, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-21T05:56:59.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 1894, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 1894, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1894, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 92.557481103, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.048868786 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-21T05:56:59.567 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1894, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1895 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 1894, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 1894, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 1146, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 746, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-09-21T05:56:59.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-21T05:56:59.569 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-21T05:56:59.570 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 78, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 7, 2024-09-21T05:56:59.571 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 6, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 6, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-21T05:56:59.572 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:56:59.573 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2143, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2853895, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2143, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2143, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2853895, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1894, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1481623, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1894, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1481623, 2024-09-21T05:56:59.575 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:59.576 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1894, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1894, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1894, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1894, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:59.577 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.578 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x555651443450": { 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:56:59.579 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: "trackedop": { 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:56:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:57:00.443 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-21T05:57:00.444 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-21T05:57:00.444 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-09-21T05:57:00.444 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:00.444 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:00.444 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-09-21T05:57:00.444 DEBUG:tasks.check_counter:Getting stats from j 2024-09-21T05:57:00.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.j perf dump 2024-09-21T05:57:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:00.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:00 smithi181 ceph-mon[34819]: pgmap v629: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:57:00.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:00 smithi028 ceph-mon[31227]: pgmap v629: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:57:00.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:00 smithi129 ceph-mon[34885]: pgmap v629: 97 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 80 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2944, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.003533477, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000047378, 2024-09-21T05:57:01.227 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002485945, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.004427426, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001475808 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T05:57:01.228 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 158, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 160, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 108000, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.014262472, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.011436219, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001294413, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:57:01.229 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 156, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.012228244, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000078386 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 108000 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.230 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 214, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 209, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 835904, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 128064, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.038054841, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.016706992, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.030980612, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:01.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 208, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.014028836, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000067446 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 835904, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 128064 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-09-21T05:57:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:01.234 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148984, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 30, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-21T05:57:01.235 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 968, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-21T05:57:01.236 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 215, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 805656, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 215, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 215, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 805656, 2024-09-21T05:57:01.237 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-21T05:57:01.238 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-21T05:57:01.947 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-09-21T05:57:01.947 DEBUG:tasks.check_counter:Getting stats from b 2024-09-21T05:57:01.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.b perf dump 2024-09-21T05:57:02.200 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:57:02.595 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:02 smithi129 ceph-mon[34885]: pgmap v630: 97 pgs: 5 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 87 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 3181, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2922, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1386085, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 3917540, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 34, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 8, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.304219162, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.122323745, 2024-09-21T05:57:02.718 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.150525930, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.038596794, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 2919, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.295991982, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000101401 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 4322, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000968672, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000224 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 4672, 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 5760 2024-09-21T05:57:02.719 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 871, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 821, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1514597, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 630701, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 12, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 6, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.096316981, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.038853736, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.055357707, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.012325014, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 822, 2024-09-21T05:57:02.720 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.065725374, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000079957 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 281, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000055346, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000196 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1069920, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 137392 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2506, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 3411, 2024-09-21T05:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1073570, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 2058998, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 16, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 7, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.283855187, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.143608362, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.172506433, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.012647452, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3409, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.463113401, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000135850 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.722 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 4050, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000944288, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000233 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 813424, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 1655344 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:02.723 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1026, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.673413149, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000656348 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 690, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.024488545, 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000035490 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.724 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 238, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 233, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 233, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 135.418334358, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.581194568 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 5, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 1, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:02.725 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 19, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 1, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 12, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 1, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 1, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 10, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 10, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 488, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 2, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 800, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 631, 2024-09-21T05:57:02.726 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 5, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 8, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 1000, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 1, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 15, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 459, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 8, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 349, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 325, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:02.727 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 4, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 234, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 6, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 325, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 73, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 73, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 252, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 33, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 524, 2024-09-21T05:57:02.728 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 463, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 1816, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 1175, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 21, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 57 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 3, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 2, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 10, 2024-09-21T05:57:02.729 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 8, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 25, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 25, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:02.730 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 15, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 1, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_started": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_write_failed": 0 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 1584, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-09-21T05:57:02.731 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 1584, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 3, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 3, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 6577616, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 0, 2024-09-21T05:57:02.732 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 362, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 434.593642631, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 1.200534924 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 0 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 16, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 388, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 372, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 14, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 113, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 99, 2024-09-21T05:57:02.733 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 12, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 525, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 513, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 261, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 261, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 129688, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 336482304 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 334, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 13, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 238, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 20, 2024-09-21T05:57:02.734 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 39, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 72, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 88.964331300, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 1.235615712 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.735 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 71, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.057774041, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000813718 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.736 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:02.737 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.739 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 51, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 46.355148943, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.908924489 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-09-21T05:57:02.740 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 14, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.004532672, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000323762 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.742 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.743 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.744 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 25, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.036547402, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001461896 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.745 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 2, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 2, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 2, 2024-09-21T05:57:02.746 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 101, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 50, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 569, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:02.747 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:02.748 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:02.749 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 297079, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 115, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:02.750 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 185, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 67928, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 268, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-09-21T05:57:02.751 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 1486, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 2484389, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 33, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 1486, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1486, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 707.679626134, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.476231242 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1486, 2024-09-21T05:57:02.752 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1624 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 1486, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 329, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 1157, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 0, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 10, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 3, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 365, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 30, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-09-21T05:57:02.753 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 638, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 326, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 20, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-09-21T05:57:02.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-09-21T05:57:02.755 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 78, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 7, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 12, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 12, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-09-21T05:57:02.756 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 174, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 2, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 56 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 1, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 25, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-09-21T05:57:02.757 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 1, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 625, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 25, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 6554, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 3263823, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 6554, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:02.758 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 6554, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 3263823, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:02.759 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1486, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 2487698, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1486, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 2487698, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:02.760 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1486, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1486, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1486, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1486, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.761 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 25, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 3325, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 25, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 3, 2024-09-21T05:57:02.762 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 3325, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x561d2ed65520": { 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 1584, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 2383312, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:02.763 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 1584, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 362, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 2383312, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "trackedop": { 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 151 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:02.764 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T05:57:02.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:02 smithi181 ceph-mon[34819]: pgmap v630: 97 pgs: 5 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 87 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:57:02.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:02 smithi028 ceph-mon[31227]: pgmap v630: 97 pgs: 5 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 87 active+clean; 16 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds.exported=15 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds.imported=8 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:03.488 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-09-21T05:57:03.488 DEBUG:tasks.check_counter:Getting stats from e 2024-09-21T05:57:03.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.e perf dump 2024-09-21T05:57:03.737 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:57:04.285 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T05:57:04.285 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 981, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 983, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 8687973, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 311968, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 7, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.120799508, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.054927738, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.028364484, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.028786874, 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:04.286 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 981, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.047415138, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000048333 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 568, 2024-09-21T05:57:04.287 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 723, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 13644749, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 539436, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 4, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.073315425, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.027807790, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.024577376, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.016513692, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 722, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.036162398, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000050086 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.288 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 360336 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 666, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 633, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 9537588, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 260746, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-09-21T05:57:04.289 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.091101120, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.033554249, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.051956882, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.013478538, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 632, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.033619714, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000053195 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1045328, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 130288 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3289, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.098728057, 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000030017 2024-09-21T05:57:04.291 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 0, 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-09-21T05:57:04.292 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 2136, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 558, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 1578, 2024-09-21T05:57:04.293 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 1339, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 2, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-09-21T05:57:04.294 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 4, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 234, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 6, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 0, 2024-09-21T05:57:04.296 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 0 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:04.297 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:04.298 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_started": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_write_failed": 0 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 11145, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-09-21T05:57:04.299 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 2, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 28, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 31320867, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 34467760, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 34464543, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 11145 2024-09-21T05:57:04.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 824, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 1940, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 1116, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 109, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 204, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 95, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 2136, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 2779, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 643, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 0, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 0, 2024-09-21T05:57:04.301 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 123316, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 325226496 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.302 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.305 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.306 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-09-21T05:57:04.306 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.306 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.306 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-09-21T05:57:04.307 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.308 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.311 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-09-21T05:57:04.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 2, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 3, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 1, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 2, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 0, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 0, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 1928, 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:04.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:04.315 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 243425, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 1044, 2024-09-21T05:57:04.317 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 185, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-09-21T05:57:04.318 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 3727493, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 49069, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 1954, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 26, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 4, 2024-09-21T05:57:04.319 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 1954, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1954, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 158.424543076, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.081077043 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1954, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1955 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 1954, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 1954, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 0, 2024-09-21T05:57:04.320 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 1165, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 786, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-09-21T05:57:04.321 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 2, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-09-21T05:57:04.322 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-09-21T05:57:04.323 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-09-21T05:57:04.324 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-09-21T05:57:04.324 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-09-21T05:57:04.324 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 78, 2024-09-21T05:57:04.324 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-09-21T05:57:04.324 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 7, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 7, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 7, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 0, 2024-09-21T05:57:04.325 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 2, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 0 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 0, 2024-09-21T05:57:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 0, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 0, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 2211, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 31655378, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 2211, 2024-09-21T05:57:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 2211, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 31655378, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:04.328 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1954, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 30277150, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1954, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 30277150, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.329 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1954, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1954, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1954, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1954, 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:04.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:04.331 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x5559924755f0": { 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:04.332 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "trackedop": { 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 0 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:04.333 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T05:57:04.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:04 smithi181 ceph-mon[34819]: pgmap v631: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:57:04.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:04 smithi028 ceph-mon[31227]: pgmap v631: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:57:04.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:04 smithi129 ceph-mon[34885]: pgmap v631: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-09-21T05:57:05.071 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-21T05:57:05.072 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-21T05:57:05.072 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-09-21T05:57:05.072 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:05.072 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:05.072 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-09-21T05:57:05.072 DEBUG:tasks.check_counter:Getting stats from h 2024-09-21T05:57:05.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.h perf dump 2024-09-21T05:57:05.319 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 727, 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 738, 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 336466, 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 2109214, 2024-09-21T05:57:05.812 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 35, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 8, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.102093553, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.033872008, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.032704239, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.022941159, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 737, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.065985489, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000089532 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 174, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000050053, 2024-09-21T05:57:05.813 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000287 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 4672, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 5760 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 3651, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 4034, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1344588, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 2269879, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 20, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 8, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.357168273, 2024-09-21T05:57:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.168842642, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.208270247, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.016943522, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 4030, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.412946332, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000102468 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 6104, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.001530855, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000250 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 559424, 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 1198592 2024-09-21T05:57:05.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 777, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 747, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1428988, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 558198, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 8, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 6, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.091967869, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.037069468, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.050400187, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.013418484, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 748, 2024-09-21T05:57:05.816 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.075606269, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000101077 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 211, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000040087, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000189 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1055264, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 139456 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-09-21T05:57:05.817 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 707, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.329312615, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000465788 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:05.818 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 768, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.028970643, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000037722 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 184, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 184, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 184, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 20.236803848, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.109982629 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.819 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 11, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 11, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 1, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 10, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 10, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 150, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-09-21T05:57:05.820 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 2, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 474, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 428, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 3, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 3, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 139, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 3, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 39, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:05.821 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 128, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 4, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 234, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 6, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 128, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 128, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 5, 2024-09-21T05:57:05.822 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 394, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 435, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 1205, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 697, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 23, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 58 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 3, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 2, 2024-09-21T05:57:05.823 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 4, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 28, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 28, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-09-21T05:57:05.824 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 3, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_started": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_write_failed": 0 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 1063, 2024-09-21T05:57:05.825 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 1063, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 3, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 3, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 5675922, 2024-09-21T05:57:05.826 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 0, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 265, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 231.594940630, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.873943172 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 0 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 15, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 163, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 148, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 13, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 26, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 13, 2024-09-21T05:57:05.827 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 11, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 189, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 178, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 77, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 77, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 125652, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 333262848 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 245, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 3, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 184, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 17, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 7, 2024-09-21T05:57:05.828 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 32, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 13.349160216, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.417161256 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 82, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 6.807658504, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.083020225 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.830 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-09-21T05:57:05.832 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.833 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 36, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.033123186, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000920088 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 6, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.003001086, 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000500181 2024-09-21T05:57:05.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.835 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.836 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.837 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-09-21T05:57:05.838 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 28, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.043860856, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001566459 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.839 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 2, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 2, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 2, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 92, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 46, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 603, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.840 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:05.841 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:05.842 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 288039, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 109, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-09-21T05:57:05.843 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 185, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 62840, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 253, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-09-21T05:57:05.844 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 1269, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 1514052, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 33, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 1269, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1269, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 492.603340694, 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.388182301 2024-09-21T05:57:05.845 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1269, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1412 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 1269, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 131, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 1138, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 0, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 10, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 3, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 268, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 28, 2024-09-21T05:57:05.846 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 712, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 128, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 20, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-09-21T05:57:05.847 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-09-21T05:57:05.848 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 78, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 7, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 12, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 12, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 174, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 0, 2024-09-21T05:57:05.849 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 69 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 1, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 25, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 1, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 700, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 28, 2024-09-21T05:57:05.850 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 5151, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 2558107, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 5151, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 5151, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 2558107, 2024-09-21T05:57:05.851 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1269, 2024-09-21T05:57:05.852 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1517776, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1269, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1517776, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:05.853 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1269, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1269, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1269, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1269, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:05.854 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 28, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 3724, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 28, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 3, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 3724, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x55e5a776d520": { 2024-09-21T05:57:05.855 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 1063, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1481618, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 1063, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 265, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1481618, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:05.856 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout: "trackedop": { 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 26 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:05.857 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds.exported=3 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:06.567 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-09-21T05:57:06.567 DEBUG:tasks.check_counter:Getting stats from k 2024-09-21T05:57:06.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.k perf dump 2024-09-21T05:57:06.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:06 smithi181 ceph-mon[34819]: pgmap v632: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:06.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:06 smithi028 ceph-mon[31227]: pgmap v632: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T05:57:06.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:06 smithi129 ceph-mon[34885]: pgmap v632: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:07.327 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T05:57:07.327 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:07.327 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 745, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 747, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 2748758, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 236974, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 6, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 2, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.081948651, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.037815430, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.018555919, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.019150076, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 745, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.032988159, 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000044279 2024-09-21T05:57:07.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 771, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 736, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1174410, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 294681, 2024-09-21T05:57:07.329 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.096940969, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.038311099, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.046820880, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.018090264, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 736, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.041206380, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000055986 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.330 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1048928, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 132496 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 617, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 776, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 207136, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 556128, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.098514345, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.045245770, 2024-09-21T05:57:07.331 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.021073338, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.024556205, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 774, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.043802800, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000056592 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 360240 2024-09-21T05:57:07.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3186, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.089901346, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000028217 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:07.333 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 0, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.334 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 39, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 39, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 39, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:07.335 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 1, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-09-21T05:57:07.336 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 0, 2024-09-21T05:57:07.337 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 0 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 0, 2024-09-21T05:57:07.338 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-09-21T05:57:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_started": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "uninline_write_failed": 0 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-09-21T05:57:07.340 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 1831, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 3, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-09-21T05:57:07.341 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 6749249, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 6748650, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 1831 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 4, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 289, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 285, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 2, 2024-09-21T05:57:07.342 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 19, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 17, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 39, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 324, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 285, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 0, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 0, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 119224, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 320745472 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 0, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-09-21T05:57:07.343 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 0, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 0, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:07.344 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:07.345 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.346 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.347 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-09-21T05:57:07.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-09-21T05:57:07.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:07.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.352 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.353 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.354 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 2, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 2, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 2, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 0, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 0, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 1929, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-09-21T05:57:07.355 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:07.356 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:07.357 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 189383, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 50, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:07.358 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 185, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 40528, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 188, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-09-21T05:57:07.359 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 1871, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 21, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 3, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 1871, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1871, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 134.265073949, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.071761129 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1871, 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1872 2024-09-21T05:57:07.360 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 1871, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 1871, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 1124, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 745, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-09-21T05:57:07.361 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 1, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-09-21T05:57:07.362 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-09-21T05:57:07.363 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 78, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 7, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 5, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 5, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 0, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 2, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 0 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-09-21T05:57:07.364 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 0, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:07.365 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 2129, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 3922642, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 2129, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 2129, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 3922642, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.366 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1871, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 2554950, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1871, 2024-09-21T05:57:07.367 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 2554950, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 1871, 2024-09-21T05:57:07.368 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1871, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1871, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1871, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:07.369 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x5582923ad380": { 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-09-21T05:57:07.370 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.371 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T05:57:07.372 INFO:teuthology.orchestra.run.smithi129.stdout: "trackedop": { 2024-09-21T05:57:07.372 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 0 2024-09-21T05:57:07.372 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T05:57:07.372 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:08.199 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-09-21T05:57:08.199 DEBUG:tasks.check_counter:Getting stats from c 2024-09-21T05:57:08.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.c perf dump 2024-09-21T05:57:08.461 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:57:08.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:08 smithi181 ceph-mon[34819]: pgmap v633: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-21T05:57:08.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:08 smithi028 ceph-mon[31227]: pgmap v633: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-21T05:57:08.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:08 smithi129 ceph-mon[34885]: pgmap v633: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-21T05:57:09.016 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 3454, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 4229, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 1366784, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 4727167, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 34, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 8, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.401870721, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.185562037, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.217605152, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.029462785, 2024-09-21T05:57:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4227, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.711061615, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000168218 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4863, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001295377, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000266 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 985536, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 1473424 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.018 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 661, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 831, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 391042, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 736442, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 10, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 7, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.090834058, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.037267835, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.022645804, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.020781183, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 829, 2024-09-21T05:57:09.019 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.079621603, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000096045 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 137, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000028597, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000208 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 1888, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 367600 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 3262, 2024-09-21T05:57:09.020 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 3419, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2287459, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 1791180, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 19, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 7, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.397376906, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.219076396, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.178427205, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.025068204, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3419, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.322102566, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000094209 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.021 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4657, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001103241, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000236 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 1070416, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 141200 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-09-21T05:57:09.022 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1147, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.676490927, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000589791 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1059, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.038662803, 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000036508 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 289, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 277, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 277, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 157.279350316, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.567795488 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 12, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 1, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 11, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 0, 2024-09-21T05:57:09.024 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 11, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 0, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 1, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 10, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 10, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 299, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 1, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 1, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 3, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 896, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 728, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 12, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 3, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 0, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 3, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 284, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 3, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 126, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 278, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 4, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 234, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 6, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 278, 2024-09-21T05:57:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 4, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 4, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 274, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 14, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 681, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 701, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 2150, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 1345, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 23, 2024-09-21T05:57:09.027 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 69 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 3, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 2, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 12, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 4, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 38, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 38, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:09.028 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 3, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_started": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_write_failed": 0 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-09-21T05:57:09.029 INFO:teuthology.orchestra.run.smithi181.stdout: "evlrg": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 1831, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 1831, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 3, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 3, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segmjr": 2, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-09-21T05:57:09.030 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 0, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 4194304, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 6749249, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 0, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 438, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 412.938539998, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.942782054 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 0 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 15, 2024-09-21T05:57:09.031 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 308, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 293, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 14, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 29, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 15, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 11, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 348, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 337, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 1, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 187, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 186, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 127496, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 334385152 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.032 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 440, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 8, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 289, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 19, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 16, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 88, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 73.431970463, 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.834454209 2024-09-21T05:57:09.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 92, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 5.687282032, 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.061818282 2024-09-21T05:57:09.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000139562, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000139562 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-09-21T05:57:09.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 54, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 78.091109250, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 1.446131652 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-09-21T05:57:09.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.003084800, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000771200 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:09.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-09-21T05:57:09.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 38, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.065764209, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001730637 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.044 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-09-21T05:57:09.044 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 2, 2024-09-21T05:57:09.044 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-09-21T05:57:09.045 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 0, 2024-09-21T05:57:09.045 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 2, 2024-09-21T05:57:09.045 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-09-21T05:57:09.045 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 121, 2024-09-21T05:57:09.045 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 60, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 616, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:09.046 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:09.047 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:09.048 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 305463, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 121, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 185, 2024-09-21T05:57:09.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 64568, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 259, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-09-21T05:57:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 1895, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 2594416, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 33, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 1895, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1895, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 726.669964864, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.383466999 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-09-21T05:57:09.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1895, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 2051 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 1895, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 292, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 1603, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 0, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 10, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 4, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 442, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 30, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-09-21T05:57:09.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 988, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 279, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 20, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-09-21T05:57:09.054 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 78, 2024-09-21T05:57:09.055 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 7, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 12, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 12, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 200, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 11, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 67 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 1, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-09-21T05:57:09.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 51, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 1, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed_ops": 976, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 38, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:09.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 7373, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 3239630, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 7373, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 7373, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 3239630, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:09.058 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 1895, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 2599175, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 1895, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2599175, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 1895, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 1895, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 1895, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 1895, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.060 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 38, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 5054, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 38, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 4, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 5054, 2024-09-21T05:57:09.061 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x55bc6437b380": { 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 1831, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 2554945, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 1831, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:09.062 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 438, 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2554945, 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "trackedop": { 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 130 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:09.063 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds.exported=3 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds.imported=3 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:09.922 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-09-21T05:57:09.922 DEBUG:tasks.check_counter:Getting stats from f 2024-09-21T05:57:09.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.f perf dump 2024-09-21T05:57:10.177 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:57:10.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:10 smithi181 ceph-mon[34819]: pgmap v634: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:10.726 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-21T05:57:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 3357, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 2774, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2322965, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 2917936, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 9, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 2, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.329822985, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.155421319, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.174523723, 2024-09-21T05:57:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.025779884, 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2779, 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.264013313, 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000095002 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4328, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001049375, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000242 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 1062992, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 141680 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 314, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 470, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 215200, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 608940, 2024-09-21T05:57:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 12, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 7, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.046651142, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.020527924, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.015404119, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.008176844, 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 469, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.057822540, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000123288 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 113, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000023436, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000207 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 362944 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 1962, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 2282, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 735381, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 1341235, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 42, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 13, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.220468968, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.097568684, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.136946357, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.011416868, 2024-09-21T05:57:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2274, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.231130751, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000101640 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3034, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000743161, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000244 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 635968, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 857424 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 659, 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.342168000, 2024-09-21T05:57:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000519223 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 503, 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.018583554, 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000036945 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-09-21T05:57:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 170, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 153, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 153, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 97.255633272, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.635657733 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 17, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 0, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 11, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 0, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 11, 2024-09-21T05:57:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 1, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 10, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 10, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 135, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 2, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 430, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 360, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 13, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 4, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 0, 2024-09-21T05:57:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 2, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 113, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 2, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 31, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 103, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 4, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 234, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 6, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 103, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 103, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 16, 2024-09-21T05:57:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 394, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 355, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 1406, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 914, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 20, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 37 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 0, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 3, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 2, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 5, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 0, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 18, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 18, 2024-09-21T05:57:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 2, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-09-21T05:57:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_started": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_write_failed": 0 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evlrg": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 1171, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 1171, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-09-21T05:57:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 3, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 3, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segmjr": 2, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 4194304, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 5780756, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 0, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 252, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 214.398079407, 2024-09-21T05:57:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.850786029 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 0 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 15, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 137, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 122, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 13, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 25, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 12, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 11, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 164, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 153, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 0, 2024-09-21T05:57:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 62, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 62, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 125252, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 332214272 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 238, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 7, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 170, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 13, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 16, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 46, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 59.060083345, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 1.283914855 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 58, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 26.720374670, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.460696115 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-09-21T05:57:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-09-21T05:57:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 29, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 11.449680859, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.394816581 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001669539, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000834769 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-09-21T05:57:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 18, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.023824859, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001323603 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 1, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 1, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 1, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-09-21T05:57:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 61, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 61, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 549, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 272407, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 103, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 185, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 62864, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 256, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 966, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 1615004, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 33, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 966, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 966, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 462.837393818, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.479127736 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 966, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1090 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 966, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 106, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 860, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 0, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 10, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 3, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 255, 2024-09-21T05:57:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 27, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 462, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 103, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 20, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-09-21T05:57:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 78, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 7, 2024-09-21T05:57:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 12, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 12, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 154, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 56 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 1, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 25, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 1, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed_ops": 450, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 18, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 5629, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 2668635, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 5629, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 5629, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2668635, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 966, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 1617398, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 966, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 1617398, 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 966, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 966, 2024-09-21T05:57:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 966, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 966, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 18, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 2394, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 18, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2394, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x557331aad5f0": { 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 1171, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 1586452, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 1171, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 252, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 1586452, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "trackedop": { 2024-09-21T05:57:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 36 2024-09-21T05:57:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:10.777 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-21T05:57:10.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:10 smithi028 ceph-mon[31227]: pgmap v634: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:10.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:10 smithi129 ceph-mon[34885]: pgmap v634: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds.exported=2 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds.imported=2 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:11.524 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-09-21T05:57:11.524 DEBUG:tasks.check_counter:Getting stats from i 2024-09-21T05:57:11.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.i perf dump 2024-09-21T05:57:11.817 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:57:12.409 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 6504, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 7200, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2497894, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 16514168, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 154, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.724595833, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.350378250, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.312056337, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.073631770, 2024-09-21T05:57:12.410 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7198, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1.896833927, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000263522 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7641, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001598235, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000209 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:12.411 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 6267, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 6231, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2492345, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 16590297, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 22, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 6, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.677439933, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.333827963, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.228706512, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.112762627, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 6232, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1.894532651, 2024-09-21T05:57:12.412 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000304000 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4639, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000969846, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000209 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 370400 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 17646, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 22527, 2024-09-21T05:57:12.413 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 8342342, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 15235496, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 19, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 8, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 1.740857796, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.853983391, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 1.690877085, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.118984051, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 22524, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 2.647598959, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000117545 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.414 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 32691, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.006387578, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000195 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 6580128, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 10340832 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.415 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 5515, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 6.014196266, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001090516 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1789, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.100368105, 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000056102 2024-09-21T05:57:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 2724, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 2716, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2716, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 475.487673554, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.175069099 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-09-21T05:57:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 8, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 7, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 2, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 414, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 1, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 1375, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 169, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 1109, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 97, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 1133, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 647, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 3, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 3, 2024-09-21T05:57:12.418 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 6, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 5046, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 4821, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 6, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 0, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 51, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 93, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 15, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 531, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 22, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 981, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:12.419 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 1910, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 4, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 234, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 6, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 1910, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 599, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 14, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 585, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 1311, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 14, 2024-09-21T05:57:12.420 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 1670, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 4211, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 6, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 9585, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 7441, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 135, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 1252 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 4, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 0, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 0, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 8, 2024-09-21T05:57:12.421 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 27, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 83, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 825, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 681, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 61, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-09-21T05:57:12.422 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 14, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 15, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-09-21T05:57:12.423 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 1, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_started": 0, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_write_failed": 0 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "evlrg": 0, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 11149, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 10448, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 10448, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 701, 2024-09-21T05:57:12.424 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 24, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 40, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 38, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 38, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 2, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segmjr": 1, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 1, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 31320867, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 34468128, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 0, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2702, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 588.819974761, 2024-09-21T05:57:12.425 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.217920049 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 0 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 829, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 1971, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 1142, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 123, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 220, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 97, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 1375, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 2892, 2024-09-21T05:57:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 1517, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 3, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 1237, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 1234, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 146316, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 399384576 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 3600, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 16, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 2724, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 37, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 67, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 765, 2024-09-21T05:57:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.925774482, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001210162 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 236, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.341904218, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001448746 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-09-21T05:57:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 61, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.048787579, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000799796 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 343, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 437.144967538, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 1.274475124 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-09-21T05:57:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000218026, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000218026 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.431 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:12.431 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.431 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.431 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 10, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001895343, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000189534 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 105, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.100311541, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000955348 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 10, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 16.281695054, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 1.628169505 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 70, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.066426256, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000948946 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 173, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1.614579631, 2024-09-21T05:57:12.433 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.009332830 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 113, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 10.709106953, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.094770857 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.434 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 99, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 5.425865789, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.054806725 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.705499630, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.176374907 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.435 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.029909292, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.004272756 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.436 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.001056321, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000264080 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 9, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.020381188, 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.002264576 2024-09-21T05:57:12.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 13, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.007795536, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000599656 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 693, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 2.061499177, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.002974746 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 2, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 3, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 1, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 2, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-09-21T05:57:12.439 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 94, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 47, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 626, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:12.440 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:12.441 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:12.442 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 1024111, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 1136, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 88, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 1, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:12.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 185, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 3278780, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 50600, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-09-21T05:57:12.444 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 7711, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 32150956, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 37, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 7711, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7711, 2024-09-21T05:57:12.445 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1375.929276154, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.178437203 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7711, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 9936 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 7711, 2024-09-21T05:57:12.446 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 1966, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 5745, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 305, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 618, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 33, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 2721, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 95, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 1, 2024-09-21T05:57:12.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 1688, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 1933, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 1236, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-09-21T05:57:12.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 6, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-09-21T05:57:12.449 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 80, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 9, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 12, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 12, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-09-21T05:57:12.450 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 1114, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 18, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 168 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 1, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 51, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-09-21T05:57:12.451 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 6, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed_ops": 1163, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 681, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 30413, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 9899563, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 30413, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:12.452 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 30413, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 9899563, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:12.453 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 7711, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 32228769, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 7711, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 32228769, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-09-21T05:57:12.454 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 7711, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 7711, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 7711, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 7711, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.455 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 681, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 77933, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 681, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 19, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 77933, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.456 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x56149eee7380": { 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 224, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 11149, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 30273824, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 11149, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 2702, 2024-09-21T05:57:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 30273600, 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "trackedop": { 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 37 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:12.458 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-21T05:57:12.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:12 smithi181 ceph-mon[34819]: pgmap v635: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:12 smithi181 ceph-mon[34819]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T05:57:12.794 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:12 smithi181 ceph-mon[34819]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T05:57:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:12 smithi028 ceph-mon[31227]: pgmap v635: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:12 smithi028 ceph-mon[31227]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T05:57:12.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:12 smithi028 ceph-mon[31227]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T05:57:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:12 smithi129 ceph-mon[34885]: pgmap v635: 97 pgs: 97 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-09-21T05:57:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:12 smithi129 ceph-mon[34885]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T05:57:12.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:12 smithi129 ceph-mon[34885]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T05:57:13.175 INFO:tasks.check_counter:Daemon mds.i mds.exported=15 2024-09-21T05:57:13.176 INFO:tasks.check_counter:Daemon mds.i mds.imported=22 2024-09-21T05:57:13.176 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-09-21T05:57:13.176 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=10 2024-09-21T05:57:13.176 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=4 2024-09-21T05:57:13.176 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=1 2024-09-21T05:57:13.176 DEBUG:tasks.check_counter:Getting stats from l 2024-09-21T05:57:13.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph daemon mds.l perf dump 2024-09-21T05:57:13.436 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T05:57:14.044 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 1921, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 1922, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2924725, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 610567, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 6, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 2, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.200729977, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.082460189, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.045324395, 2024-09-21T05:57:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.060848041, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1920, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.085456508, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000044508 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 3, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 164, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 716, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 362954, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 4, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 3, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.011988495, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.009023072, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.001406661, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.000077993, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 162, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.009931864, 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000061307 2024-09-21T05:57:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 362256 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 266, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 231, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 1051068, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 134446, 2024-09-21T05:57:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 3, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 2, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.043240581, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.018042756, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.034575185, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.000031716, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 231, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.015934219, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000068979 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 1050800, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 134128 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3290, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.091115536, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000027694 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 0, 2024-09-21T05:57:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 26, 2024-09-21T05:57:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 26, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 26, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 1, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-09-21T05:57:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T05:57:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 0, 2024-09-21T05:57:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 0 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-09-21T05:57:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_path": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_quiesce_inode": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0, 2024-09-21T05:57:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_started": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_succeeded": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "uninline_write_failed": 0 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evlrg": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 1584, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 0, 2024-09-21T05:57:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 0, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 0, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 3, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "segmjr": 2, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 0, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 4194304, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 6577616, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 6573047, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 1584 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 4, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 378, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 374, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 2, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 105, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 103, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 26, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 508, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 482, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 0, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 0, 2024-09-21T05:57:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 119156, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 320749568 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-09-21T05:57:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 2, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 0, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 2, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-09-21T05:57:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 1942, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T05:57:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-09-21T05:57:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 181320, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 47, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-09-21T05:57:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8376, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 185, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 31688, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 136, 2024-09-21T05:57:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 1923, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 21, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 3, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 1923, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1923, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 107.407750934, 2024-09-21T05:57:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.055854264 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1923, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1924 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 1923, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 1923, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 0, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 1156, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 0, 2024-09-21T05:57:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 765, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 1, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-09-21T05:57:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-09-21T05:57:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 80, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 9, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 4, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 4, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-09-21T05:57:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 2, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 0 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 0, 2024-09-21T05:57:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed_ops": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 2186, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 3763520, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 2186, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 2186, 2024-09-21T05:57:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 3763520, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 1923, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 2383317, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 1923, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2383317, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-09-21T05:57:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 1923, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 1923, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 1923, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 1923, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x55acf3521380": { 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-09-21T05:57:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "trackedop": { 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 0 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-21T05:57:14.090 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-21T05:57:14.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:14 smithi181 ceph-mon[34819]: pgmap v638: 97 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 76 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T05:57:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:14 smithi028 ceph-mon[31227]: pgmap v638: 97 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 76 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T05:57:14.821 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-09-21T05:57:14.821 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-21T05:57:14.831 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-21T05:57:14.831 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:57:14.831 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T05:57:14.849 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-21T05:57:14.849 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T05:57:14.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:14 smithi129 ceph-mon[34885]: pgmap v638: 97 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 76 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T05:57:14.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T05:57:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 -- 172.21.15.28:0/2610429134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4103b60 msgr2=0x7f22a4103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2610429134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4103b60 0x7f22a4103f70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2294009900 tx=0x7f229402a410 comp rx=0 tx=0).stop 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 -- 172.21.15.28:0/2610429134 shutdown_connections 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2610429134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a4109df0 0x7f22a4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2610429134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a41094a0 0x7f22a41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2610429134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4103b60 0x7f22a4103f70 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 -- 172.21.15.28:0/2610429134 >> 172.21.15.28:0/2610429134 conn(0x7f22a407b850 msgr2=0x7f22a407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.711+0000 7f22a91c0640 1 -- 172.21.15.28:0/2610429134 shutdown_connections 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.712+0000 7f22a91c0640 1 -- 172.21.15.28:0/2610429134 wait complete. 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.712+0000 7f22a91c0640 1 Processor -- start 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 -- start start 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a41094a0 0x7f22a4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 0x7f22a4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4113ab0 con 0x7f22a4109df0 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4113930 con 0x7f22a41094a0 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a91c0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4113c30 con 0x7f22a4103b60 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 0x7f22a4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a41094a0 0x7f22a4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35820/0 (socket says 172.21.15.28:35820) 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 -- 172.21.15.28:0/2855594133 learned_addr learned my addr 172.21.15.28:0/2855594133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 -- 172.21.15.28:0/2855594133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a41094a0 msgr2=0x7f22a4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a41094a0 0x7f22a4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 -- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 msgr2=0x7f22a4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 0x7f22a4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a2d9d640 1 -- 172.21.15.28:0/2855594133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a41a8760 con 0x7f22a4103b60 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.713+0000 7f22a359e640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 0x7f22a4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f22a2d9d640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2294005940 tx=0x7f2294004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229403f030 con 0x7f22a4103b60 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a41a89f0 con 0x7f22a4103b60 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2294003aa0 con 0x7f22a4103b60 2024-09-21T05:57:15.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294035070 con 0x7f22a4103b60 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.714+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a41a8ed0 con 0x7f22a4103b60 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.715+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a4103f70 con 0x7f22a4103b60 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.717+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f2294042030 con 0x7f22a4103b60 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.717+0000 7f228bfff640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 0x7f228c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.717+0000 7f22a259c640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 0x7f228c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.718+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8572+0+0 (secure 0 0 0) 0x7f2294108100 con 0x7f22a4103b60 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.718+0000 7f22a259c640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 0x7f228c0b8b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f22a4109270 tx=0x7f229800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:15.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.720+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f22940d29a0 con 0x7f22a4103b60 2024-09-21T05:57:15.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.938+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f22a402e640 con 0x7f22a4103b60 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.939+0000 7f228bfff640 1 -- 172.21.15.28:0/2855594133 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v80) ==== 81+0+886 (secure 0 0 0) 0x7f22940d78e0 con 0x7f22a4103b60 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3351178737 2024-09-22T05:38:12.622156+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1190896102 2024-09-22T05:38:12.622156+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2589278469 2024-09-22T05:36:43.853731+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2066463820 2024-09-22T05:36:43.853731+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3235284089 2024-09-22T05:36:43.853731+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/689076330 2024-09-22T05:37:11.040387+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3969089930 2024-09-22T05:37:11.040387+0000 2024-09-21T05:57:15.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1887508254 2024-09-22T05:37:11.040387+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1617434937 2024-09-22T05:38:12.622156+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2299789749 2024-09-22T05:36:43.853731+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/689076330 2024-09-22T05:37:11.040387+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4032693217 2024-09-22T05:37:11.040387+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3235284089 2024-09-22T05:36:43.853731+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2040548036 2024-09-22T05:38:12.622156+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2040548036 2024-09-22T05:38:12.622156+0000 2024-09-21T05:57:15.942 INFO:teuthology.orchestra.run.smithi028.stderr:listed 15 entries 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 msgr2=0x7f228c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 0x7f228c0b8b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f22a4109270 tx=0x7f229800b390 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 msgr2=0x7f22a4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2294005940 tx=0x7f2294004270 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 shutdown_connections 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f228c0b6680 0x7f228c0b8b90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a4109df0 0x7f22a4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22a41094a0 0x7f22a4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 --2- 172.21.15.28:0/2855594133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22a4103b60 0x7f22a4075710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 >> 172.21.15.28:0/2855594133 conn(0x7f22a407b850 msgr2=0x7f22a4107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 shutdown_connections 2024-09-21T05:57:15.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:15.942+0000 7f22a91c0640 1 -- 172.21.15.28:0/2855594133 wait complete. 2024-09-21T05:57:16.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:16 smithi028 ceph-mon[31227]: pgmap v639: 97 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 62 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-21T05:57:16.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:16 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2855594133' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T05:57:16.639 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-21T05:57:16.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:57:16.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-21T05:57:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:16 smithi129 ceph-mon[34885]: pgmap v639: 97 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 62 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-21T05:57:16.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:16 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2855594133' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T05:57:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:16 smithi181 ceph-mon[34819]: pgmap v639: 97 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 62 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-21T05:57:17.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:16 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2855594133' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T05:57:18.050 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T05:57:18.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:57:18.050 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-21T05:57:18.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:57:18.068 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-21T05:57:18.124 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-21T05:57:18.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:57:18.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T05:57:18.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T05:57:18.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-21T05:57:18.295 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T05:57:18.313 INFO:tasks.cephadm:Teardown begin 2024-09-21T05:57:18.313 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-21T05:57:18.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:57:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:18 smithi028 ceph-mon[31227]: pgmap v640: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 127 B/s wr, 9 op/s 2024-09-21T05:57:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:18.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:18 smithi129 ceph-mon[34885]: pgmap v640: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 127 B/s wr, 9 op/s 2024-09-21T05:57:19.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:18 smithi181 ceph-mon[34819]: pgmap v640: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 127 B/s wr, 9 op/s 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 -- 172.21.15.28:0/1765381230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c1094b0 msgr2=0x7f8e8c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 --2- 172.21.15.28:0/1765381230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c1098c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009900 tx=0x7f8e8002a410 comp rx=0 tx=0).stop 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 -- 172.21.15.28:0/1765381230 shutdown_connections 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 --2- 172.21.15.28:0/1765381230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c109e00 0x7f8e8c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 --2- 172.21.15.28:0/1765381230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c1098c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 --2- 172.21.15.28:0/1765381230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.122+0000 7f8e9276c640 1 -- 172.21.15.28:0/1765381230 >> 172.21.15.28:0/1765381230 conn(0x7f8e8c077c60 msgr2=0x7f8e8c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.123+0000 7f8e9276c640 1 -- 172.21.15.28:0/1765381230 shutdown_connections 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.123+0000 7f8e9276c640 1 -- 172.21.15.28:0/1765381230 wait complete. 2024-09-21T05:57:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 Processor -- start 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 -- start start 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c109e00 0x7f8e8c103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c113ac0 con 0x7f8e8c101f70 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c113940 con 0x7f8e8c109e00 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.124+0000 7f8e9276c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c113c40 con 0x7f8e8c1094b0 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43530/0 (socket says 172.21.15.28:43530) 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 -- 172.21.15.28:0/3248583452 learned_addr learned my addr 172.21.15.28:0/3248583452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e90d09640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c109e00 0x7f8e8c103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8b7fe640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 -- 172.21.15.28:0/3248583452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 msgr2=0x7f8e8c103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 -- 172.21.15.28:0/3248583452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c109e00 msgr2=0x7f8e8c103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c109e00 0x7f8e8c103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 -- 172.21.15.28:0/3248583452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c1a8730 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8b7fe640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.125+0000 7f8e8bfff640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00e9f0 tx=0x7f8e7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.126+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00cd80 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.126+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e7c0108e0 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.126+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c018980 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.126+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c1a8a20 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.126+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c1a8f30 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.127+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e8c102380 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.129+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f8e7c010430 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.130+0000 7f8e897fa640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 0x7f8e740b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.130+0000 7f8e8b7fe640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 0x7f8e740b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.130+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8572+0+0 (secure 0 0 0) 0x7f8e7c0e8010 con 0x7f8e8c101f70 2024-09-21T05:57:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.131+0000 7f8e8b7fe640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 0x7f8e740b8b90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009230 tx=0x7f8e80000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:19.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.133+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8e7c0ae530 con 0x7f8e8c101f70 2024-09-21T05:57:19.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.374+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e8c02e640 con 0x7f8e8c101f70 2024-09-21T05:57:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.374+0000 7f8e897fa640 1 -- 172.21.15.28:0/3248583452 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11986 (secure 0 0 0) 0x7f8e7c0b2910 con 0x7f8e8c101f70 2024-09-21T05:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:57:19.377 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"btime":"2024-09-21T05:56:39:464552+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:56:39.280658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:57:19.377 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 54 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.377+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 msgr2=0x7f8e740b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.377+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 0x7f8e740b8b90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009230 tx=0x7f8e80000bc0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.377+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 msgr2=0x7f8e8c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.377+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00e9f0 tx=0x7f8e7c00eeb0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 shutdown_connections 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e8c109e00 0x7f8e8c103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e8c1094b0 0x7f8e8c103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f8e740b6680 0x7f8e740b8b90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 --2- 172.21.15.28:0/3248583452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c101f70 0x7f8e8c102f10 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 >> 172.21.15.28:0/3248583452 conn(0x7f8e8c077c60 msgr2=0x7f8e8c1054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 shutdown_connections 2024-09-21T05:57:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:19.378+0000 7f8e9276c640 1 -- 172.21.15.28:0/3248583452 wait complete. 2024-09-21T05:57:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:19 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3248583452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:19.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:19 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3248583452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:20.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:19 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3248583452' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:20.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T05:57:20.315 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:20.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:20 smithi028 ceph-mon[31227]: pgmap v641: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:57:20.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:20 smithi181 ceph-mon[34819]: pgmap v641: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:57:20.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:20 smithi129 ceph-mon[34885]: pgmap v641: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T05:57:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 -- 172.21.15.28:0/3031472561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e241094a0 msgr2=0x7f9e241098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 --2- 172.21.15.28:0/3031472561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e241094a0 0x7f9e241098b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009900 tx=0x7f9e1802a410 comp rx=0 tx=0).stop 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 -- 172.21.15.28:0/3031472561 shutdown_connections 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 --2- 172.21.15.28:0/3031472561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e24109df0 0x7f9e24110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 --2- 172.21.15.28:0/3031472561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e241094a0 0x7f9e241098b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 --2- 172.21.15.28:0/3031472561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e24103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 -- 172.21.15.28:0/3031472561 >> 172.21.15.28:0/3031472561 conn(0x7f9e2407b850 msgr2=0x7f9e2407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 -- 172.21.15.28:0/3031472561 shutdown_connections 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.933+0000 7f9e28899640 1 -- 172.21.15.28:0/3031472561 wait complete. 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 Processor -- start 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 -- start start 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e2419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 0x7f9e2419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e24113ab0 con 0x7f9e24109df0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e24113930 con 0x7f9e24103b60 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e28899640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e24113c30 con 0x7f9e241094a0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e2259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e2419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43544/0 (socket says 172.21.15.28:43544) 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 -- 172.21.15.28:0/1055104116 learned_addr learned my addr 172.21.15.28:0/1055104116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e21d9b640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 0x7f9e2419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 -- 172.21.15.28:0/1055104116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 msgr2=0x7f9e2419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 0x7f9e2419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 -- 172.21.15.28:0/1055104116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 msgr2=0x7f9e2419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e2419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e22d9d640 1 -- 172.21.15.28:0/1055104116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e241a04d0 con 0x7f9e24109df0 2024-09-21T05:57:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e21d9b640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 0x7f9e2419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.934+0000 7f9e2259c640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e2419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e22d9d640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00eaa0 tx=0x7f9e1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00cdb0 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e1c005020 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e241a07c0 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0116d0 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.935+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e241a6f20 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.936+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e24103f70 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.938+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f9e1c0051d0 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.938+0000 7f9e137fe640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 0x7f9e0c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.938+0000 7f9e2259c640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 0x7f9e0c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.938+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8572+0+0 (secure 0 0 0) 0x7f9e1c0e7c70 con 0x7f9e24109df0 2024-09-21T05:57:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.939+0000 7f9e2259c640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 0x7f9e0c0b8b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009fc0 tx=0x7f9e14005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:20.942+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9e1c016030 con 0x7f9e24109df0 2024-09-21T05:57:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.190+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e2419b690 con 0x7f9e24109df0 2024-09-21T05:57:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.190+0000 7f9e137fe640 1 -- 172.21.15.28:0/1055104116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11986 (secure 0 0 0) 0x7f9e2419b690 con 0x7f9e24109df0 2024-09-21T05:57:21.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:57:21.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"btime":"2024-09-21T05:56:39:464552+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14784,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2759619114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2759619114},{"type":"v1","addr":"172.21.15.28:6835","nonce":2759619114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14796,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/783262834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":783262834},{"type":"v1","addr":"172.21.15.28:6837","nonce":783262834}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T05:44:39.761973+0000","modified":"2024-09-21T05:56:39.280658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24688,"mds_1":24680,"mds_2":24673,"mds_3":24664,"mds_4":24653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14850":{"gid":14850,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/2924711217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":2924711217},{"type":"v1","addr":"172.21.15.28:6839","nonce":2924711217}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14856":{"gid":14856,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2765835727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2765835727},{"type":"v1","addr":"172.21.15.28:6841","nonce":2765835727}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24614":{"gid":24614,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6833/2193748712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2193748712},{"type":"v1","addr":"172.21.15.129:6833","nonce":2193748712}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24622":{"gid":24622,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6833/1938433457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1938433457},{"type":"v1","addr":"172.21.15.181:6833","nonce":1938433457}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2842004697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2842004697},{"type":"v1","addr":"172.21.15.129:6835","nonce":2842004697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24653":{"gid":24653,"name":"h","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.129:6837/2283221342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2283221342},{"type":"v1","addr":"172.21.15.129:6837","nonce":2283221342}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24664":{"gid":24664,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.181:6835/1316004847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1316004847},{"type":"v1","addr":"172.21.15.181:6835","nonce":1316004847}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24673":{"gid":24673,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.181:6837/2388771333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2388771333},{"type":"v1","addr":"172.21.15.181:6837","nonce":2388771333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/3631196486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":3631196486},{"type":"v1","addr":"172.21.15.129:6839","nonce":3631196486}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.181:6839/3430071220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3430071220},{"type":"v1","addr":"172.21.15.181:6839","nonce":3430071220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24688,"qdb_cluster":[24653,24664,24673,24680,24688]},"id":1}]} 2024-09-21T05:57:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 54 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 msgr2=0x7f9e0c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 0x7f9e0c0b8b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009fc0 tx=0x7f9e14005f90 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 msgr2=0x7f9e2419fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00eaa0 tx=0x7f9e1c00ef60 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 shutdown_connections 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f9e0c0b6680 0x7f9e0c0b8b90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e24109df0 0x7f9e2419fda0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e241094a0 0x7f9e2419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 --2- 172.21.15.28:0/1055104116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e24103b60 0x7f9e2419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 >> 172.21.15.28:0/1055104116 conn(0x7f9e2407b850 msgr2=0x7f9e24105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 shutdown_connections 2024-09-21T05:57:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:21.193+0000 7f9e28899640 1 -- 172.21.15.28:0/1055104116 wait complete. 2024-09-21T05:57:21.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:21 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1055104116' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:21.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T05:57:21.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:21 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1055104116' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:22.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:21 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1055104116' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T05:57:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 -- 172.21.15.28:0/118927375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec8105330 msgr2=0x7f2ec810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/118927375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec8105330 0x7f2ec810d880 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009900 tx=0x7f2eb802a410 comp rx=0 tx=0).stop 2024-09-21T05:57:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 -- 172.21.15.28:0/118927375 shutdown_connections 2024-09-21T05:57:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/118927375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec810de40 0x7f2ec8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/118927375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec8105330 0x7f2ec810d880 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/118927375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 -- 172.21.15.28:0/118927375 >> 172.21.15.28:0/118927375 conn(0x7f2ec8079730 msgr2=0x7f2ec8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.662+0000 7f2ecfc53640 1 -- 172.21.15.28:0/118927375 shutdown_connections 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.663+0000 7f2ecfc53640 1 -- 172.21.15.28:0/118927375 wait complete. 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.663+0000 7f2ecfc53640 1 Processor -- start 2024-09-21T05:57:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.663+0000 7f2ecfc53640 1 -- start start 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.663+0000 7f2ecfc53640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.663+0000 7f2ecfc53640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecfc53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 0x7f2ec81a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecfc53640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8113aa0 con 0x7f2ec810de40 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecfc53640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8113920 con 0x7f2ec81049e0 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecfc53640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8113c20 con 0x7f2ec8105330 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd9ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ece1f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 0x7f2ec81a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35864/0 (socket says 172.21.15.28:35864) 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 -- 172.21.15.28:0/2133468958 learned_addr learned my addr 172.21.15.28:0/2133468958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd9ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:59392/0 (socket says 172.21.15.28:59392) 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 -- 172.21.15.28:0/2133468958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 msgr2=0x7f2ec819c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec819c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 -- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 msgr2=0x7f2ec81a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 0x7f2ec81a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 -- 172.21.15.28:0/2133468958 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec81a25e0 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ece1f0640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 0x7f2ec81a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.664+0000 7f2ecd1ee640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2eb802a930 tx=0x7f2eb8031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.665+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb803f030 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.665+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec81a2870 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.665+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eb8004490 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.665+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8035070 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.665+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81a9030 con 0x7f2ec8105330 2024-09-21T05:57:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.666+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec806c240 con 0x7f2ec8105330 2024-09-21T05:57:22.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.667+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7f2eb8042030 con 0x7f2ec8105330 2024-09-21T05:57:22.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.668+0000 7f2ebeffd640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 0x7f2eb40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:22.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.668+0000 7f2ecd9ef640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 0x7f2eb40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:22.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.668+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8572+0+0 (secure 0 0 0) 0x7f2eb8107e40 con 0x7f2ec8105330 2024-09-21T05:57:22.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.669+0000 7f2ecd9ef640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 0x7f2eb40b8b90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009bd0 tx=0x7f2ec4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.672+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2eb80d2720 con 0x7f2ec8105330 2024-09-21T05:57:22.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:22 smithi028 ceph-mon[31227]: pgmap v642: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:22.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:22 smithi129 ceph-mon[34885]: pgmap v642: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:22.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.873+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ec8104df0 con 0x7f2ec8105330 2024-09-21T05:57:22.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.875+0000 7f2ebeffd640 1 -- 172.21.15.28:0/2133468958 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+23279 (secure 0 0 0) 0x7f2ec8104df0 con 0x7f2ec8105330 2024-09-21T05:57:22.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-21T05:57:22.877 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"fsid":"42500114-77db-11ef-baf5-efdc52797490","created":"2024-09-21T05:36:12.179132+0000","modified":"2024-09-21T05:57:12.262704+0000","last_up_change":"2024-09-21T05:43:52.146560+0000","last_in_change":"2024-09-21T05:43:28.547856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T05:40:11.086256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T05:44:35.312867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T05:44:37.712198+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aebf5461-2659-41f2-8c87-f7d8bce2cc0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6803","nonce":3566179718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6805","nonce":3566179718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6809","nonce":3566179718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3566179718},{"type":"v1","addr":"172.21.15.28:6807","nonce":3566179718}]},"public_addr":"172.21.15.28:6803/3566179718","cluster_addr":"172.21.15.28:6805/3566179718","heartbeat_back_addr":"172.21.15.28:6809/3566179718","heartbeat_front_addr":"172.21.15.28:6807/3566179718","state":["exists","up"]},{"osd":1,"uuid":"ced7a484-b9ec-49ce-825a-16be7ba41ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6801","nonce":1623835479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6803","nonce":1623835479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6807","nonce":1623835479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1623835479},{"type":"v1","addr":"172.21.15.129:6805","nonce":1623835479}]},"public_addr":"172.21.15.129:6801/1623835479","cluster_addr":"172.21.15.129:6803/1623835479","heartbeat_back_addr":"172.21.15.129:6807/1623835479","heartbeat_front_addr":"172.21.15.129:6805/1623835479","state":["exists","up"]},{"osd":2,"uuid":"0efbfaff-07a3-4354-b453-5b1ee7fea224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6801","nonce":3260464300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6803","nonce":3260464300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6807","nonce":3260464300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3260464300},{"type":"v1","addr":"172.21.15.181:6805","nonce":3260464300}]},"public_addr":"172.21.15.181:6801/3260464300","cluster_addr":"172.21.15.181:6803/3260464300","heartbeat_back_addr":"172.21.15.181:6807/3260464300","heartbeat_front_addr":"172.21.15.181:6805/3260464300","state":["exists","up"]},{"osd":3,"uuid":"7e76bace-e04c-43a5-95d6-753d39f5adb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6811","nonce":2300237806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6813","nonce":2300237806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6817","nonce":2300237806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2300237806},{"type":"v1","addr":"172.21.15.28:6815","nonce":2300237806}]},"public_addr":"172.21.15.28:6811/2300237806","cluster_addr":"172.21.15.28:6813/2300237806","heartbeat_back_addr":"172.21.15.28:6817/2300237806","heartbeat_front_addr":"172.21.15.28:6815/2300237806","state":["exists","up"]},{"osd":4,"uuid":"e34f6ec9-ba14-4098-af52-2c147d72cba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6809","nonce":645644998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6811","nonce":645644998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6815","nonce":645644998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":645644998},{"type":"v1","addr":"172.21.15.129:6813","nonce":645644998}]},"public_addr":"172.21.15.129:6809/645644998","cluster_addr":"172.21.15.129:6811/645644998","heartbeat_back_addr":"172.21.15.129:6815/645644998","heartbeat_front_addr":"172.21.15.129:6813/645644998","state":["exists","up"]},{"osd":5,"uuid":"76455120-2b9e-4fe9-9ae7-3f85cf2a54c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6809","nonce":1579505960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6811","nonce":1579505960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6815","nonce":1579505960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1579505960},{"type":"v1","addr":"172.21.15.181:6813","nonce":1579505960}]},"public_addr":"172.21.15.181:6809/1579505960","cluster_addr":"172.21.15.181:6811/1579505960","heartbeat_back_addr":"172.21.15.181:6815/1579505960","heartbeat_front_addr":"172.21.15.181:6813/1579505960","state":["exists","up"]},{"osd":6,"uuid":"7a460509-dbe4-4d67-a88a-d45cf3c21227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6819","nonce":88277262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6821","nonce":88277262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6825","nonce":88277262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":88277262},{"type":"v1","addr":"172.21.15.28:6823","nonce":88277262}]},"public_addr":"172.21.15.28:6819/88277262","cluster_addr":"172.21.15.28:6821/88277262","heartbeat_back_addr":"172.21.15.28:6825/88277262","heartbeat_front_addr":"172.21.15.28:6823/88277262","state":["exists","up"]},{"osd":7,"uuid":"9ed1f260-f7ce-4489-80a1-043099519502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6817","nonce":317230964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6819","nonce":317230964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6823","nonce":317230964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":317230964},{"type":"v1","addr":"172.21.15.129:6821","nonce":317230964}]},"public_addr":"172.21.15.129:6817/317230964","cluster_addr":"172.21.15.129:6819/317230964","heartbeat_back_addr":"172.21.15.129:6823/317230964","heartbeat_front_addr":"172.21.15.129:6821/317230964","state":["exists","up"]},{"osd":8,"uuid":"2a7581c5-4c13-4d96-82a4-c910a21c857c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6817","nonce":3198301680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6819","nonce":3198301680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6823","nonce":3198301680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3198301680},{"type":"v1","addr":"172.21.15.181:6821","nonce":3198301680}]},"public_addr":"172.21.15.181:6817/3198301680","cluster_addr":"172.21.15.181:6819/3198301680","heartbeat_back_addr":"172.21.15.181:6823/3198301680","heartbeat_front_addr":"172.21.15.181:6821/3198301680","state":["exists","up"]},{"osd":9,"uuid":"0e472f0b-cc9d-4f4f-90d2-a9fd8a4296f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6827","nonce":2894928742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6829","nonce":2894928742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6833","nonce":2894928742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2894928742},{"type":"v1","addr":"172.21.15.28:6831","nonce":2894928742}]},"public_addr":"172.21.15.28:6827/2894928742","cluster_addr":"172.21.15.28:6829/2894928742","heartbeat_back_addr":"172.21.15.28:6833/2894928742","heartbeat_front_addr":"172.21.15.28:6831/2894928742","state":["exists","up"]},{"osd":10,"uuid":"8d6e3073-eab4-4ac9-a926-c42fa4e501d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6825","nonce":3703525783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6827","nonce":3703525783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6831","nonce":3703525783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3703525783},{"type":"v1","addr":"172.21.15.129:6829","nonce":3703525783}]},"public_addr":"172.21.15.129:6825/3703525783","cluster_addr":"172.21.15.129:6827/3703525783","heartbeat_back_addr":"172.21.15.129:6831/3703525783","heartbeat_front_addr":"172.21.15.129:6829/3703525783","state":["exists","up"]},{"osd":11,"uuid":"f7d4e141-618d-41ae-a3e8-4120e69bc428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6825","nonce":3814712780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6827","nonce":3814712780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6831","nonce":3814712780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3814712780},{"type":"v1","addr":"172.21.15.181:6829","nonce":3814712780}]},"public_addr":"172.21.15.181:6825/3814712780","cluster_addr":"172.21.15.181:6827/3814712780","heartbeat_back_addr":"172.21.15.181:6831/3814712780","heartbeat_front_addr":"172.21.15.181:6829/3814712780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:19.217596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:39:43.595990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:08.089131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:34.304081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:40:59.545535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:24.547696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:41:48.866905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:13.576030+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:42:37.207679+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:01.723785+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:25.360921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T05:43:49.730768+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3351178737":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/1190896102":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2589278469":"2024-09-22T05:36:43.853731+0000","172.21.15.28:0/2066463820":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/3969089930":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1887508254":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/1617434937":"2024-09-22T05:38:12.622156+0000","172.21.15.28:0/2299789749":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6801/689076330":"2024-09-22T05:37:11.040387+0000","172.21.15.28:0/4032693217":"2024-09-22T05:37:11.040387+0000","172.21.15.28:6800/3235284089":"2024-09-22T05:36:43.853731+0000","172.21.15.28:6800/2040548036":"2024-09-22T05:38:12.622156+0000","172.21.15.28:6801/2040548036":"2024-09-22T05:38:12.622156+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 msgr2=0x7f2eb40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 0x7f2eb40b8b90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4009bd0 tx=0x7f2ec4005f90 comp rx=0 tx=0).stop 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 msgr2=0x7f2ec819cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2eb802a930 tx=0x7f2eb8031a20 comp rx=0 tx=0).stop 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 shutdown_connections 2024-09-21T05:57:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7f2eb40b6680 0x7f2eb40b8b90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec810de40 0x7f2ec81a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ec8105330 0x7f2ec819cc10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 --2- 172.21.15.28:0/2133468958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec81049e0 0x7f2ec819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 >> 172.21.15.28:0/2133468958 conn(0x7f2ec8079730 msgr2=0x7f2ec8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 shutdown_connections 2024-09-21T05:57:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:22.877+0000 7f2ecfc53640 1 -- 172.21.15.28:0/2133468958 wait complete. 2024-09-21T05:57:23.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:22 smithi181 ceph-mon[34819]: pgmap v642: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:23.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs fail cephfs 2024-09-21T05:57:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:23.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:23 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2133468958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:57:23.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:23 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2133468958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:57:24.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:23 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2133468958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T05:57:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/2130178090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 msgr2=0x7ff0d8103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/2130178090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d8103f90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009900 tx=0x7ff0c802a410 comp rx=0 tx=0).stop 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/2130178090 shutdown_connections 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/2130178090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 0x7ff0d8110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/2130178090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d81098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/2130178090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d8103f90 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/2130178090 >> 172.21.15.28:0/2130178090 conn(0x7ff0d80ff610 msgr2=0x7ff0d8101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/2130178090 shutdown_connections 2024-09-21T05:57:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.271+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/2130178090 wait complete. 2024-09-21T05:57:24.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.272+0000 7ff0dcc1c640 1 Processor -- start 2024-09-21T05:57:24.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 -- start start 2024-09-21T05:57:24.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 0x7ff0d819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8113ad0 con 0x7ff0d8103b80 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8113950 con 0x7ff0d8109e10 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0dcc1c640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8113c50 con 0x7ff0d81094c0 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0cffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0d6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 0x7ff0d819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43584/0 (socket says 172.21.15.28:43584) 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0d659c640 1 -- 172.21.15.28:0/1584871349 learned_addr learned my addr 172.21.15.28:0/1584871349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.273+0000 7ff0cffff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.28:35870/0 (socket says 172.21.15.28:35870) 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 -- 172.21.15.28:0/1584871349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 msgr2=0x7ff0d819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 -- 172.21.15.28:0/1584871349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 msgr2=0x7ff0d819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 0x7ff0d819fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 -- 172.21.15.28:0/1584871349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d81a0430 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0cffff640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0d659c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005940 tx=0x7ff0c80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c803f030 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d81a06c0 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0c80044c0 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8035070 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.274+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d80755f0 con 0x7ff0d8103b80 2024-09-21T05:57:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.275+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d8104e00 con 0x7ff0d8103b80 2024-09-21T05:57:24.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.278+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff0c8042030 con 0x7ff0d8103b80 2024-09-21T05:57:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.279+0000 7ff0cf7fe640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 0x7ff0c40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.279+0000 7ff0cffff640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 0x7ff0c40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.279+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8572+0+0 (secure 0 0 0) 0x7ff0c8108140 con 0x7ff0d8103b80 2024-09-21T05:57:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.279+0000 7ff0cffff640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 0x7ff0c40b8b90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8108e50 tx=0x7ff0c0014040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.283+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff0c80d29e0 con 0x7ff0d8103b80 2024-09-21T05:57:24.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.526+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff0d8103f90 con 0x7ff0d8103b80 2024-09-21T05:57:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.528+0000 7ff0cf7fe640 1 -- 172.21.15.28:0/1584871349 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) ==== 161+0+0 (secure 0 0 0) 0x7ff0c80d7920 con 0x7ff0d8103b80 2024-09-21T05:57:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.531+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 msgr2=0x7ff0c40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.531+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 0x7ff0c40b8b90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8108e50 tx=0x7ff0c0014040 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.531+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 msgr2=0x7ff0d819a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.531+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005940 tx=0x7ff0c80040c0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 shutdown_connections 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0d8109e10 0x7ff0d819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0d81094c0 0x7ff0d819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff0c40b6680 0x7ff0c40b8b90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 --2- 172.21.15.28:0/1584871349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d8103b80 0x7ff0d819a4c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.532+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 >> 172.21.15.28:0/1584871349 conn(0x7ff0d80ff610 msgr2=0x7ff0d8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.533+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 shutdown_connections 2024-09-21T05:57:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:24.533+0000 7ff0dcc1c640 1 -- 172.21.15.28:0/1584871349 wait complete. 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mds[74334]: -- 172.21.15.28:0/1771737726 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf56835860 con 0x55cf5a9dd180 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: starting mds.a at 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a[31223]: 2024-09-21T05:57:24.527+0000 7f6fe11c3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: pgmap v643: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:24.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T05:57:24.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T05:57:24.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T05:57:24.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T05:57:24.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T05:57:24.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:24 smithi028 ceph-mon[31227]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T05:57:24.864 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e[62883]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.864 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e[62883]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.864 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mds[62887]: -- 172.21.15.129:0/1750082935 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55741d91d860 con 0x557421ac5600 2024-09-21T05:57:24.864 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e[62883]: starting mds.e at 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h[63155]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h[63155]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mds[63159]: -- 172.21.15.129:0/1647655228 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d647397860 con 0x55d64b53f180 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h[63155]: starting mds.h at 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k[62619]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k[62619]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mds[62623]: -- 172.21.15.129:0/1640725875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55effe2dfc20 con 0x55f002487180 2024-09-21T05:57:24.865 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k[62619]: starting mds.k at 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mds[63440]: -- 172.21.15.129:0/676966024 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5637a4e21860 con 0x5637a8fc9180 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: starting mds.b at 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: pgmap v643: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T05:57:24.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T05:57:24.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T05:57:24.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T05:57:24.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:24 smithi129 ceph-mon[34885]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T05:57:24.944 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.944 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.944 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: starting mds.f at 2024-09-21T05:57:24.944 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: pgmap v643: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/1584871349' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mon[34819]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mds[64277]: -- 172.21.15.181:0/276578280 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b3cf657860 con 0x55b3d37ff600 2024-09-21T05:57:24.945 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: starting mds.c at 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mds[64549]: -- 172.21.15.181:0/3400402317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c490181860 con 0x55c494329180 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: starting mds.i at 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: ignoring --setuser ceph since I am not root 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: ignoring --setgroup ceph since I am not root 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mds[63740]: -- 172.21.15.181:0/1969370505 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c5c908ea0 con 0x555c60aa7a80 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-mds[63740]: -- 172.21.15.181:0/1969370505 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c5c8ff860 con 0x555c60aa7a80 2024-09-21T05:57:24.946 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 05:57:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: starting mds.l at 2024-09-21T05:57:25.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-21T05:57:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/1783487364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb0102a60 msgr2=0x7fbfb0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/1783487364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb0102a60 0x7fbfb0102e70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009ca0 tx=0x7fbfa002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/1783487364 shutdown_connections 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/1783487364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 0x7fbfb010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/1783487364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb01083a0 0x7fbfb01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/1783487364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb0102a60 0x7fbfb0102e70 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/1783487364 >> 172.21.15.28:0/1783487364 conn(0x7fbfb0079730 msgr2=0x7fbfb0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/1783487364 shutdown_connections 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.143+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/1783487364 wait complete. 2024-09-21T05:57:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 Processor -- start 2024-09-21T05:57:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 -- start start 2024-09-21T05:57:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 0x7fbfb019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 0x7fbfb019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0113ab0 con 0x7fbfb01083a0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0113930 con 0x7fbfb0102a60 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfb5cd9640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0113c30 con 0x7fbfb0108cf0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfaeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfaf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 0x7fbfb019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfaeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59376/0 (socket says 172.21.15.28:59376) 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfaeffd640 1 -- 172.21.15.28:0/2183460982 learned_addr learned my addr 172.21.15.28:0/2183460982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.144+0000 7fbfaffff640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 0x7fbfb019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 -- 172.21.15.28:0/2183460982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 msgr2=0x7fbfb019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 0x7fbfb019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 -- 172.21.15.28:0/2183460982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 msgr2=0x7fbfb019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 0x7fbfb019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 -- 172.21.15.28:0/2183460982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb01a03b0 con 0x7fbfb01083a0 2024-09-21T05:57:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaf7fe640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 0x7fbfb019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfaeffd640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9c0 tx=0x7fbf9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800cd80 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf980054d0 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb01a06a0 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98010430 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.145+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb01a6e00 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.146+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb0102e70 con 0x7fbfb01083a0 2024-09-21T05:57:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.148+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fbf98004fb0 con 0x7fbfb01083a0 2024-09-21T05:57:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.149+0000 7fbfacff9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 0x7fbf9c0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.149+0000 7fbfaf7fe640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 0x7fbf9c0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.149+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8958+0+0 (secure 0 0 0) 0x7fbf980e7ed0 con 0x7fbfb01083a0 2024-09-21T05:57:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.149+0000 7fbfaf7fe640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 0x7fbf9c0b8c40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006010 tx=0x7fbfa0004a40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.152+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbf98056710 con 0x7fbfb01083a0 2024-09-21T05:57:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.394+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbfb002e640 con 0x7fbfb01083a0 2024-09-21T05:57:26.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: pgmap v645: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:26.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6833/1984286218,v1:172.21.15.129:6835/1984286218] up:boot 2024-09-21T05:57:26.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6832/2797949058,v1:172.21.15.129:6834/2797949058] up:boot 2024-09-21T05:57:26.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6836/4036132981,v1:172.21.15.129:6837/4036132981] up:boot 2024-09-21T05:57:26.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6833/3995044298,v1:172.21.15.181:6835/3995044298] up:boot 2024-09-21T05:57:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.589+0000 7fbfacff9640 1 -- 172.21.15.28:0/2183460982 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) ==== 104+0+0 (secure 0 0 0) 0x7fbf980ae210 con 0x7fbfb01083a0 2024-09-21T05:57:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 msgr2=0x7fbf9c0b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 0x7fbf9c0b8c40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006010 tx=0x7fbfa0004a40 comp rx=0 tx=0).stop 2024-09-21T05:57:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 msgr2=0x7fbfb019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9c0 tx=0x7fbf9800ee80 comp rx=0 tx=0).stop 2024-09-21T05:57:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 shutdown_connections 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfb0108cf0 0x7fbfb019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fbf9c0b6730 0x7fbf9c0b8c40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb01083a0 0x7fbfb019a9e0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 --2- 172.21.15.28:0/2183460982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb0102a60 0x7fbfb019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 >> 172.21.15.28:0/2183460982 conn(0x7fbfb0079730 msgr2=0x7fbfb0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 shutdown_connections 2024-09-21T05:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:26.591+0000 7fbfb5cd9640 1 -- 172.21.15.28:0/2183460982 wait complete. 2024-09-21T05:57:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: pgmap v645: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6833/1984286218,v1:172.21.15.129:6835/1984286218] up:boot 2024-09-21T05:57:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6832/2797949058,v1:172.21.15.129:6834/2797949058] up:boot 2024-09-21T05:57:26.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6836/4036132981,v1:172.21.15.129:6837/4036132981] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6833/3995044298,v1:172.21.15.181:6835/3995044298] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6832/2812314592,v1:172.21.15.181:6834/2812314592] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.129:6838/4097528281,v1:172.21.15.129:6839/4097528281] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6836/3142999175,v1:172.21.15.181:6837/3142999175] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.181:6838/1757311368,v1:172.21.15.181:6839/1757311368] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6838/2666677290,v1:172.21.15.28:6840/2666677290] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: mds.? [v2:172.21.15.28:6839/3509856610,v1:172.21.15.28:6841/3509856610] up:boot 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:57:26.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:57:26.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:57:26.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:57:26.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:57:26.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:57:26.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:26 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:57:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6832/2812314592,v1:172.21.15.181:6834/2812314592] up:boot 2024-09-21T05:57:26.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.129:6838/4097528281,v1:172.21.15.129:6839/4097528281] up:boot 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6836/3142999175,v1:172.21.15.181:6837/3142999175] up:boot 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.181:6838/1757311368,v1:172.21.15.181:6839/1757311368] up:boot 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6838/2666677290,v1:172.21.15.28:6840/2666677290] up:boot 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: mds.? [v2:172.21.15.28:6839/3509856610,v1:172.21.15.28:6841/3509856610] up:boot 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:57:26.865 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:57:26.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:57:26.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:57:26.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:57:26.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:57:26.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:26 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:57:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: pgmap v645: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T05:57:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6833/1984286218,v1:172.21.15.129:6835/1984286218] up:boot 2024-09-21T05:57:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6832/2797949058,v1:172.21.15.129:6834/2797949058] up:boot 2024-09-21T05:57:27.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6836/4036132981,v1:172.21.15.129:6837/4036132981] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6833/3995044298,v1:172.21.15.181:6835/3995044298] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6832/2812314592,v1:172.21.15.181:6834/2812314592] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.129:6838/4097528281,v1:172.21.15.129:6839/4097528281] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6836/3142999175,v1:172.21.15.181:6837/3142999175] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.181:6838/1757311368,v1:172.21.15.181:6839/1757311368] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6838/2666677290,v1:172.21.15.28:6840/2666677290] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: mds.? [v2:172.21.15.28:6839/3509856610,v1:172.21.15.28:6841/3509856610] up:boot 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T05:57:27.045 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T05:57:27.046 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T05:57:27.046 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T05:57:27.046 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T05:57:27.046 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='mgr.14223 172.21.15.28:0/1365515996' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T05:57:27.046 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:26 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T05:57:27.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-21T05:57:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:27.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:27 smithi028 ceph-mon[31227]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T05:57:27.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:27 smithi028 ceph-mon[31227]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T05:57:27.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:27 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:57:27.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:27 smithi028 ceph-mon[31227]: fsmap 12 up:standby 2024-09-21T05:57:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:27 smithi129 ceph-mon[34885]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T05:57:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:27 smithi129 ceph-mon[34885]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T05:57:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:27 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:57:27.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:27 smithi129 ceph-mon[34885]: fsmap 12 up:standby 2024-09-21T05:57:28.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:27 smithi181 ceph-mon[34819]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T05:57:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:27 smithi181 ceph-mon[34819]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T05:57:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:27 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2183460982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T05:57:28.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:27 smithi181 ceph-mon[34819]: fsmap 12 up:standby 2024-09-21T05:57:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3075193494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c102a60 msgr2=0x7ff59c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3075193494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c102a60 0x7ff59c102e70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009900 tx=0x7ff58c02a3f0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3075193494 shutdown_connections 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3075193494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c108cf0 0x7ff59c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3075193494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 0x7ff59c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3075193494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c102a60 0x7ff59c102e70 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3075193494 >> 172.21.15.28:0/3075193494 conn(0x7ff59c079730 msgr2=0x7ff59c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3075193494 shutdown_connections 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.105+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3075193494 wait complete. 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 Processor -- start 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 -- start start 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c102a60 0x7ff59c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 0x7ff59c19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c112960 con 0x7ff59c108cf0 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c1127e0 con 0x7ff59c1083a0 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.106+0000 7ff5a0cd0640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c112ae0 con 0x7ff59c102a60 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59386/0 (socket says 172.21.15.28:59386) 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 -- 172.21.15.28:0/3104867456 learned_addr learned my addr 172.21.15.28:0/3104867456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff599d9b640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 0x7ff59c19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59a59c640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c102a60 0x7ff59c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 -- 172.21.15.28:0/3104867456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c102a60 msgr2=0x7ff59c19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c102a60 0x7ff59c19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 -- 172.21.15.28:0/3104867456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 msgr2=0x7ff59c19b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 0x7ff59c19b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 -- 172.21.15.28:0/3104867456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c1a6490 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff59ad9d640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff59400cce0 tx=0x7ff59400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59400f450 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59400faa0 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59c1a6780 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594018b70 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.107+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59c1a6c70 con 0x7ff59c108cf0 2024-09-21T05:57:28.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.108+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59c102e70 con 0x7ff59c108cf0 2024-09-21T05:57:28.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.110+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7ff594018d10 con 0x7ff59c108cf0 2024-09-21T05:57:28.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.111+0000 7ff58b7fe640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 0x7ff5840b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:28.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.111+0000 7ff59a59c640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 0x7ff5840b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:28.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.111+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8958+0+0 (secure 0 0 0) 0x7ff5940e7e00 con 0x7ff59c108cf0 2024-09-21T05:57:28.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.111+0000 7ff59a59c640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 0x7ff5840b8bc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000c00 tx=0x7ff58c031400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.114+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff594014030 con 0x7ff59c108cf0 2024-09-21T05:57:28.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.326+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff59c06b270 con 0x7ff59c108cf0 2024-09-21T05:57:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.591+0000 7ff58b7fe640 1 -- 172.21.15.28:0/3104867456 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) ==== 181+0+0 (secure 0 0 0) 0x7ff5940ae1d0 con 0x7ff59c108cf0 2024-09-21T05:57:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-21T05:57:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 msgr2=0x7ff5840b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 0x7ff5840b8bc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000c00 tx=0x7ff58c031400 comp rx=0 tx=0).stop 2024-09-21T05:57:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 msgr2=0x7ff59c19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff59400cce0 tx=0x7ff59400a620 comp rx=0 tx=0).stop 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 shutdown_connections 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7ff5840b66b0 0x7ff5840b8bc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff59c108cf0 0x7ff59c19bd10 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff59c1083a0 0x7ff59c19b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 --2- 172.21.15.28:0/3104867456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff59c102a60 0x7ff59c19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.594+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 >> 172.21.15.28:0/3104867456 conn(0x7ff59c079730 msgr2=0x7ff59c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.595+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 shutdown_connections 2024-09-21T05:57:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:28.595+0000 7ff5a0cd0640 1 -- 172.21.15.28:0/3104867456 wait complete. 2024-09-21T05:57:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:28 smithi129 ceph-mon[34885]: pgmap v646: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:28.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:28 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:28 smithi028 ceph-mon[31227]: pgmap v646: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:28.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:28 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:29.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:28 smithi181 ceph-mon[34819]: pgmap v646: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-21T05:57:29.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:28 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:29.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 42500114-77db-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-21T05:57:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T05:57:29.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:29 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:29.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:29 smithi028 ceph-mon[31227]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T05:57:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:29 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:29.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:29 smithi129 ceph-mon[34885]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T05:57:30.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:29 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/3104867456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:30.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:29 smithi181 ceph-mon[34819]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 -- 172.21.15.28:0/2828210918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 msgr2=0x7fb13810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2828210918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13810f520 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fb130009ca0 tx=0x7fb13002a8c0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 -- 172.21.15.28:0/2828210918 shutdown_connections 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2828210918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13810f520 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2828210918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1381083a0 0x7fb1381087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2828210918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb138102a60 0x7fb138102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 -- 172.21.15.28:0/2828210918 >> 172.21.15.28:0/2828210918 conn(0x7fb138079730 msgr2=0x7fb138079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 -- 172.21.15.28:0/2828210918 shutdown_connections 2024-09-21T05:57:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.089+0000 7fb13e80a640 1 -- 172.21.15.28:0/2828210918 wait complete. 2024-09-21T05:57:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 Processor -- start 2024-09-21T05:57:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 -- start start 2024-09-21T05:57:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 0x7fb13819b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb138112960 con 0x7fb138108cf0 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1381127e0 con 0x7fb1381083a0 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.090+0000 7fb13e80a640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb138112ae0 con 0x7fb138102a60 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb13cda7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1377fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 0x7fb13819b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb13cda7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59410/0 (socket says 172.21.15.28:59410) 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1377fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.28:34700/0 (socket says 172.21.15.28:34700) 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb13cda7640 1 -- 172.21.15.28:0/2454689212 learned_addr learned my addr 172.21.15.28:0/2454689212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 -- 172.21.15.28:0/2454689212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 msgr2=0x7fb13819b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 -- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 msgr2=0x7fb13819bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 -- 172.21.15.28:0/2454689212 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb138199000 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1377fe640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb13cda7640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb137fff640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 0x7fb13819b210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9f0 tx=0x7fb12800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12800cd80 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1381992f0 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1280035f0 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.091+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128010690 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.092+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb138199880 con 0x7fb138102a60 2024-09-21T05:57:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.093+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb138102e70 con 0x7fb138102a60 2024-09-21T05:57:30.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.094+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) ==== 154643+0+0 (secure 0 0 0) 0x7fb1280037a0 con 0x7fb138102a60 2024-09-21T05:57:30.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.095+0000 7fb1357fa640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 0x7fb1200b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T05:57:30.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.095+0000 7fb1377fe640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 0x7fb1200b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T05:57:30.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.095+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8375+0+0 (secure 0 0 0) 0x7fb1280e7ab0 con 0x7fb138102a60 2024-09-21T05:57:30.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.095+0000 7fb1377fe640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 0x7fb1200b8b90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb138105f20 tx=0x7fb12c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T05:57:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.098+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb1280ae0c0 con 0x7fb138102a60 2024-09-21T05:57:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.291+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb13806b2d0 con 0x7fb138102a60 2024-09-21T05:57:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.613+0000 7fb1357fa640 1 -- 172.21.15.28:0/2454689212 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) ==== 169+0+0 (secure 0 0 0) 0x7fb1280b24a0 con 0x7fb138102a60 2024-09-21T05:57:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-21T05:57:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.617+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 msgr2=0x7fb1200b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.617+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 0x7fb1200b8b90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb138105f20 tx=0x7fb12c005f90 comp rx=0 tx=0).stop 2024-09-21T05:57:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 msgr2=0x7fb13819b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 0x7fb13819b210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9f0 tx=0x7fb12800eeb0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 shutdown_connections 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:6800/4211147350,v1:172.21.15.28:6801/4211147350] conn(0x7fb1200b6680 0x7fb1200b8b90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb138108cf0 0x7fb13819bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1381083a0 0x7fb13819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 --2- 172.21.15.28:0/2454689212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb138102a60 0x7fb13819b210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T05:57:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.618+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 >> 172.21.15.28:0/2454689212 conn(0x7fb138079730 msgr2=0x7fb1381043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T05:57:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.619+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 shutdown_connections 2024-09-21T05:57:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-21T05:57:30.619+0000 7fb13e80a640 1 -- 172.21.15.28:0/2454689212 wait complete. 2024-09-21T05:57:31.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:30 smithi028 ceph-mon[31227]: pgmap v648: 65 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail 2024-09-21T05:57:31.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:30 smithi028 ceph-mon[31227]: from='client.? 172.21.15.28:0/2454689212' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:30 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:30 smithi181 ceph-mon[34819]: pgmap v648: 65 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail 2024-09-21T05:57:31.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:30 smithi181 ceph-mon[34819]: from='client.? 172.21.15.28:0/2454689212' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:30 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:30 smithi129 ceph-mon[34885]: pgmap v648: 65 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 active+clean; 16 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail 2024-09-21T05:57:31.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:30 smithi129 ceph-mon[34885]: from='client.? 172.21.15.28:0/2454689212' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:30 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T05:57:31.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:57:31.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:57:31.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T05:57:31.736 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T05:57:31.736 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T05:57:31.765 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T05:57:31.779 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T05:57:31.795 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T05:57:31.795 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T05:57:31.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a 2024-09-21T05:57:31.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:31 smithi028 ceph-mon[31227]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:31.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:31 smithi028 ceph-mon[31227]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T05:57:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:31 smithi181 ceph-mon[34819]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:32.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:31 smithi181 ceph-mon[34819]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T05:57:32.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:31 smithi129 ceph-mon[34885]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T05:57:32.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:31 smithi129 ceph-mon[34885]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T05:57:32.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:31 smithi028 systemd[1]: Stopping Ceph mon.a for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:32.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a[31223]: 2024-09-21T05:57:32.245+0000 7f6fe69ce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:57:32.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a[31223]: 2024-09-21T05:57:32.245+0000 7f6fe69ce640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T05:57:32.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:32 smithi028 podman[110511]: 2024-09-21 05:57:32.481474801 +0000 UTC m=+0.342728033 container died a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:57:33.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:33 smithi028 podman[110511]: 2024-09-21 05:57:33.00317312 +0000 UTC m=+0.864426335 container cleanup a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2) 2024-09-21T05:57:33.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:33 smithi028 bash[110511]: ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a 2024-09-21T05:57:33.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 21 05:57:33 smithi028 podman[110523]: 2024-09-21 05:57:33.263479559 +0000 UTC m=+0.779851256 container remove a73d3a9255f0e68f7eaf1180d77bf62221bf15edbce4562c3b97c0268e7bc1af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T05:57:33.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.a.service' 2024-09-21T05:57:33.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:33.688 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T05:57:33.689 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-21T05:57:33.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mon.b 2024-09-21T05:57:34.009 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:33 smithi129 systemd[1]: Stopping Ceph mon.b for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b[34881]: 2024-09-21T05:57:34.006+0000 7fdc1ec16640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:57:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b[34881]: 2024-09-21T05:57:34.006+0000 7fdc1ec16640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T05:57:34.364 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 podman[66999]: 2024-09-21 05:57:34.02745555 +0000 UTC m=+0.150204835 container died d1186cbf6d5bdfbe3157847d5fd643e655c9d96e8a75aa1710fb35c99c0e6a8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T05:57:34.651 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 podman[66999]: 2024-09-21 05:57:34.456752926 +0000 UTC m=+0.579502211 container cleanup d1186cbf6d5bdfbe3157847d5fd643e655c9d96e8a75aa1710fb35c99c0e6a8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:57:34.651 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 bash[66999]: ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b 2024-09-21T05:57:34.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.b.service' 2024-09-21T05:57:34.907 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 podman[67011]: 2024-09-21 05:57:34.649502052 +0000 UTC m=+0.621892288 container remove d1186cbf6d5bdfbe3157847d5fd643e655c9d96e8a75aa1710fb35c99c0e6a8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-b, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T05:57:34.907 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T05:57:34.907 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 systemd[1]: Stopped Ceph mon.b for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:57:34.907 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 05:57:34 smithi129 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mon.b.service: Consumed 13.781s CPU time. 2024-09-21T05:57:34.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:34.938 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-21T05:57:34.938 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-21T05:57:34.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mon.c 2024-09-21T05:57:35.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:34 smithi181 systemd[1]: Stopping Ceph mon.c for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:35.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c[34815]: 2024-09-21T05:57:35.242+0000 7f4e0911c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:57:35.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c[34815]: 2024-09-21T05:57:35.242+0000 7f4e0911c640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T05:57:35.544 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 podman[67934]: 2024-09-21 05:57:35.374737273 +0000 UTC m=+0.240494169 container died 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T05:57:35.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 podman[67934]: 2024-09-21 05:57:35.709933366 +0000 UTC m=+0.575690305 container cleanup 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T05:57:35.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 bash[67934]: ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c 2024-09-21T05:57:36.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mon.c.service' 2024-09-21T05:57:36.121 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:35 smithi181 podman[67948]: 2024-09-21 05:57:35.869517004 +0000 UTC m=+0.496792315 container remove 509a1ea6af6eb50b6d9e859063bac1ed5778f6eda7e01624fe276c52df78f9a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T05:57:36.121 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:36 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T05:57:36.121 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:36 smithi181 systemd[1]: Stopped Ceph mon.c for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T05:57:36.121 INFO:journalctl@ceph.mon.c.smithi181.stdout:Sep 21 05:57:36 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mon.c.service: Consumed 14.187s CPU time. 2024-09-21T05:57:36.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:36.145 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-21T05:57:36.145 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T05:57:36.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x 2024-09-21T05:57:36.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:57:36 smithi028 systemd[1]: Stopping Ceph mgr.x for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:36.971 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:57:36 smithi028 podman[110626]: 2024-09-21 05:57:36.66291552 +0000 UTC m=+0.228921557 container died 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:57:37.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:57:37 smithi028 podman[110626]: 2024-09-21 05:57:37.175917932 +0000 UTC m=+0.741923978 container cleanup 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T05:57:37.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:57:37 smithi028 bash[110626]: ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x 2024-09-21T05:57:37.811 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 21 05:57:37 smithi028 podman[110636]: 2024-09-21 05:57:37.461588769 +0000 UTC m=+0.794845000 container remove 4efa3b0812743208877341962a608b8bd3b56c8395c309ad71c194f3954b348b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-x, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T05:57:37.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-21T05:57:37.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:37.942 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T05:57:37.942 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T05:57:37.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.y 2024-09-21T05:57:38.250 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:37 smithi129 systemd[1]: Stopping Ceph mgr.y for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:38.553 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:38 smithi129 podman[67124]: 2024-09-21 05:57:38.24873483 +0000 UTC m=+0.115425884 container died 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T05:57:38.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:38 smithi129 podman[67124]: 2024-09-21 05:57:38.552108102 +0000 UTC m=+0.418799167 container cleanup 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T05:57:38.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:38 smithi129 bash[67124]: ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y 2024-09-21T05:57:38.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:38 smithi129 podman[67136]: 2024-09-21 05:57:38.795640596 +0000 UTC m=+0.541214086 container remove 83a052978e297805acb540b3fbbdecdc41d779f362120c89678e558c2c52e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T05:57:38.864 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Sep 21 05:57:38 smithi129 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T05:57:38.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-21T05:57:39.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:39.007 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T05:57:39.007 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-21T05:57:39.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.z 2024-09-21T05:57:39.294 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 systemd[1]: Stopping Ceph mgr.z for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:39.648 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 podman[68056]: 2024-09-21 05:57:39.351993848 +0000 UTC m=+0.159533962 container died 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T05:57:39.994 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 podman[68056]: 2024-09-21 05:57:39.647485731 +0000 UTC m=+0.455025845 container cleanup 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T05:57:39.994 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 bash[68056]: ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z 2024-09-21T05:57:39.994 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 podman[68067]: 2024-09-21 05:57:39.882481257 +0000 UTC m=+0.525144655 container remove 3dfa4497666fd814ef964b0ae29eaf20d812ebbf8d5a1c8bb330c6de5e1c9f06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mgr-z, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T05:57:39.995 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Sep 21 05:57:39 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-09-21T05:57:40.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mgr.z.service' 2024-09-21T05:57:40.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:40.042 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-21T05:57:40.042 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T05:57:40.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.0 2024-09-21T05:57:40.459 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:40 smithi028 systemd[1]: Stopping Ceph osd.0 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:40.811 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:57:40.456+0000 7f77e3a2a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:57:40.811 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:57:40.456+0000 7f77e3a2a640 -1 osd.0 83 *** Got signal Terminated *** 2024-09-21T05:57:40.811 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0[48189]: 2024-09-21T05:57:40.456+0000 7f77e3a2a640 -1 osd.0 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:57:48.008 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:47 smithi028 podman[110743]: 2024-09-21 05:57:47.574487756 +0000 UTC m=+7.242803730 container died e22c7ed3a590d97b12c704fde22fa43b1055b8633a8431f58a5c2ae73e9398a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T05:57:48.259 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 podman[110743]: 2024-09-21 05:57:48.006663454 +0000 UTC m=+7.674979427 container cleanup e22c7ed3a590d97b12c704fde22fa43b1055b8633a8431f58a5c2ae73e9398a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T05:57:48.260 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 bash[110743]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0 2024-09-21T05:57:48.561 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 podman[110760]: 2024-09-21 05:57:48.258383435 +0000 UTC m=+0.685692558 container remove e22c7ed3a590d97b12c704fde22fa43b1055b8633a8431f58a5c2ae73e9398a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T05:57:48.997 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 podman[110845]: 2024-09-21 05:57:48.610014541 +0000 UTC m=+0.020463357 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:57:48.997 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 podman[110845]: 2024-09-21 05:57:48.743080184 +0000 UTC m=+0.153528988 container create 623a279970b0c24f344fd2d4fbd1c16ca0421351d0dfc0485f9975b608f9429d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True) 2024-09-21T05:57:49.311 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:48 smithi028 podman[110845]: 2024-09-21 05:57:48.995633073 +0000 UTC m=+0.406081877 container init 623a279970b0c24f344fd2d4fbd1c16ca0421351d0dfc0485f9975b608f9429d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True) 2024-09-21T05:57:49.311 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:49 smithi028 podman[110845]: 2024-09-21 05:57:49.000503088 +0000 UTC m=+0.410951901 container start 623a279970b0c24f344fd2d4fbd1c16ca0421351d0dfc0485f9975b608f9429d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T05:57:49.312 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:49 smithi028 podman[110845]: 2024-09-21 05:57:49.087417839 +0000 UTC m=+0.497866636 container attach 623a279970b0c24f344fd2d4fbd1c16ca0421351d0dfc0485f9975b608f9429d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:57:49.312 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 21 05:57:49 smithi028 podman[110845]: 2024-09-21 05:57:49.199884608 +0000 UTC m=+0.610333415 container died 623a279970b0c24f344fd2d4fbd1c16ca0421351d0dfc0485f9975b608f9429d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD) 2024-09-21T05:57:49.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.0.service' 2024-09-21T05:57:49.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:57:49.850 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T05:57:49.850 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T05:57:49.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.1 2024-09-21T05:57:50.342 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:57:49 smithi129 systemd[1]: Stopping Ceph osd.1 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:57:50.614 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:57:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:57:50.338+0000 7f36bf082640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:57:50.614 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:57:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:57:50.338+0000 7f36bf082640 -1 osd.1 83 *** Got signal Terminated *** 2024-09-21T05:57:50.614 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:57:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1[39507]: 2024-09-21T05:57:50.338+0000 7f36bf082640 -1 osd.1 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:58:20.614 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:20 smithi129 bash[67248]: time="2024-09-21T05:58:20Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1 in 30 seconds, resorting to SIGKILL" 2024-09-21T05:58:20.614 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:20 smithi129 podman[67248]: 2024-09-21 05:58:20.526952551 +0000 UTC m=+30.488420782 container died 54bcc27ac7951c4cfcadf28aad7d77801a084386079b20c41d1a5cea9744961a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T05:58:21.185 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:20 smithi129 podman[67248]: 2024-09-21 05:58:20.971560907 +0000 UTC m=+30.933029129 container cleanup 54bcc27ac7951c4cfcadf28aad7d77801a084386079b20c41d1a5cea9744961a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T05:58:21.185 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:20 smithi129 bash[67248]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1 2024-09-21T05:58:21.611 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 podman[67265]: 2024-09-21 05:58:21.184393596 +0000 UTC m=+0.651831399 container remove 54bcc27ac7951c4cfcadf28aad7d77801a084386079b20c41d1a5cea9744961a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T05:58:21.611 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@osd.1.service: Main process exited, code=exited, status=137/n/a 2024-09-21T05:58:21.864 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 podman[67357]: 2024-09-21 05:58:21.512653194 +0000 UTC m=+0.017180091 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:58:21.864 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 podman[67357]: 2024-09-21 05:58:21.632387688 +0000 UTC m=+0.136914573 container create a077a558c3fb2a3fad70fb11b3c38ad5a5ef8804ad348880714f642483c598c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T05:58:22.148 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 podman[67357]: 2024-09-21 05:58:21.951680955 +0000 UTC m=+0.456207841 container init a077a558c3fb2a3fad70fb11b3c38ad5a5ef8804ad348880714f642483c598c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:58:22.149 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:21 smithi129 podman[67357]: 2024-09-21 05:58:21.9551233 +0000 UTC m=+0.459650181 container start a077a558c3fb2a3fad70fb11b3c38ad5a5ef8804ad348880714f642483c598c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T05:58:22.149 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:22 smithi129 podman[67357]: 2024-09-21 05:58:22.026681133 +0000 UTC m=+0.531208029 container attach a077a558c3fb2a3fad70fb11b3c38ad5a5ef8804ad348880714f642483c598c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T05:58:22.456 INFO:journalctl@ceph.osd.1.smithi129.stdout:Sep 21 05:58:22 smithi129 podman[67357]: 2024-09-21 05:58:22.147543965 +0000 UTC m=+0.652070841 container died a077a558c3fb2a3fad70fb11b3c38ad5a5ef8804ad348880714f642483c598c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T05:58:22.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.1.service' 2024-09-21T05:58:22.794 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:58:22.794 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T05:58:22.794 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T05:58:22.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.2 2024-09-21T05:58:23.101 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:22 smithi181 systemd[1]: Stopping Ceph osd.2 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:58:23.372 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:58:23.098+0000 7f543b497640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:58:23.729 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:58:23.098+0000 7f543b497640 -1 osd.2 83 *** Got signal Terminated *** 2024-09-21T05:58:23.730 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2[39920]: 2024-09-21T05:58:23.098+0000 7f543b497640 -1 osd.2 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:58:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:47.756+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:48.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:48 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:48.411+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:48.804+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:49.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:49.363+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:49.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:49.302+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:50.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:49.614+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:50.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:49.825+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:50.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:49.879+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:50.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:50.383+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:50.562 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:50.253+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:51.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:50.632+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:51.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:50.818+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:51.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:50.904+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:51.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:51.402+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:51.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:51.215+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:52.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:51.631+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:51.830+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:52.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:51.939+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:52.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:52.222+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:52.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:52.420+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:53.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:52.587+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:53.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:52.861+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:53.194 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 bash[68178]: time="2024-09-21T05:58:53Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2 in 30 seconds, resorting to SIGKILL" 2024-09-21T05:58:53.194 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:52.925+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:53.465 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:53.192+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:53.492 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:53.307+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:53.492 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 podman[68178]: 2024-09-21 05:58:53.273822172 +0000 UTC m=+30.295866106 container died 55ead999486c46b29468f59c86ca8c9d1248c7d8d0a84285e36b8c7e0e5082bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T05:58:53.757 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 podman[68178]: 2024-09-21 05:58:53.663661225 +0000 UTC m=+30.685705156 container cleanup 55ead999486c46b29468f59c86ca8c9d1248c7d8d0a84285e36b8c7e0e5082bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T05:58:53.757 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 bash[68178]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2 2024-09-21T05:58:53.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:53.606+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:53.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:53.464+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:54.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:53.949+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:54.044 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 podman[68205]: 2024-09-21 05:58:53.923717917 +0000 UTC m=+0.644120302 container remove 55ead999486c46b29468f59c86ca8c9d1248c7d8d0a84285e36b8c7e0e5082bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T05:58:54.044 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:53 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@osd.2.service: Main process exited, code=exited, status=137/n/a 2024-09-21T05:58:54.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:53.863+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:54.469 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:54.164+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:54.520 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:54.261+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:54.520 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.291455811 +0000 UTC m=+0.100596785 container create c8abce55d2cf90264b1818ddd1989a1aa0acaab3cadcfea316974ca6154fdf83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T05:58:54.520 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.20735528 +0000 UTC m=+0.016496270 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:58:54.520 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.518930024 +0000 UTC m=+0.328071008 container init c8abce55d2cf90264b1818ddd1989a1aa0acaab3cadcfea316974ca6154fdf83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T05:58:54.794 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.522784616 +0000 UTC m=+0.331925601 container start c8abce55d2cf90264b1818ddd1989a1aa0acaab3cadcfea316974ca6154fdf83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T05:58:54.794 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.577380024 +0000 UTC m=+0.386521007 container attach c8abce55d2cf90264b1818ddd1989a1aa0acaab3cadcfea316974ca6154fdf83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=) 2024-09-21T05:58:54.794 INFO:journalctl@ceph.osd.2.smithi181.stdout:Sep 21 05:58:54 smithi181 podman[68302]: 2024-09-21 05:58:54.741648869 +0000 UTC m=+0.550789843 container died c8abce55d2cf90264b1818ddd1989a1aa0acaab3cadcfea316974ca6154fdf83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:58:54.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:54.613+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:54.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:54.468+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:55.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:54.854+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:55.283 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:54.952+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:55.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.2.service' 2024-09-21T05:58:55.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:58:55.346 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T05:58:55.346 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T05:58:55.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.3 2024-09-21T05:58:55.472 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:55 smithi028 systemd[1]: Stopping Ceph osd.3 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:58:55.472 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:55.190+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:55.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:55.281+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:55.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:55.585+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:55.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:55.470+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:55.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:55.682+0000 7ffbbff26640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:58:55.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:55.682+0000 7ffbbff26640 -1 osd.3 83 *** Got signal Terminated *** 2024-09-21T05:58:55.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:55.682+0000 7ffbbff26640 -1 osd.3 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:58:56.241 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:55.927+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:56.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:55.870+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:56.485 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:56.160+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:56.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:56.239+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:56.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:56.582+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:56.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:56.483+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:57.277 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:56.878+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:57.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:56.894+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:57.481 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:57.196+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:57.544 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:57.275+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:57.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:57.480+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:57.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:57.609+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:58.282 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:57.870+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:58.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:57.908+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:58.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:58.280+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:58.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:58.481+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:58:58.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:58.242+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:59.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:58.613+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:59.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:58.901+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:58:59.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:58.884+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:58:59.525 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:58:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:58:59.246+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:58:59.793 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:58:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:58:59.309+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:58:59.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:58:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:58:59.615+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:58:59.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:58:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:58:59.523+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:59:00.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:58:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:58:59.876+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:00.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:58:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:58:59.898+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:00.537 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:00.254+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:00.793 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:00.325+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:00.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3[54741]: 2024-09-21T05:59:00.534+0000 7ffbbc511640 -1 osd.3 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.861413+0000 front 2024-09-21T05:57:44.861204+0000 (oldest deadline 2024-09-21T05:58:48.261246+0000) 2024-09-21T05:59:00.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:00.617+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:00.849+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:01.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:00.831+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:01.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:01.242+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:01.793 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:01.296+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:02.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:01.632+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:02.061 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:01 smithi028 podman[110946]: 2024-09-21 05:59:01.694748366 +0000 UTC m=+6.138360505 container died 0295a955ff3ed4629fcfda24c3c9b99ca9ea68fbaf122e7c6e60112581e2972c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T05:59:02.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:01.806+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:02.249 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:01.807+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:02.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:02.194+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:02.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:02 smithi028 podman[110946]: 2024-09-21 05:59:02.299097882 +0000 UTC m=+6.742710021 container cleanup 0295a955ff3ed4629fcfda24c3c9b99ca9ea68fbaf122e7c6e60112581e2972c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T05:59:02.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:02 smithi028 bash[110946]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3 2024-09-21T05:59:02.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:02.247+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:02.811 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:02 smithi028 podman[110962]: 2024-09-21 05:59:02.53431179 +0000 UTC m=+0.832384157 container remove 0295a955ff3ed4629fcfda24c3c9b99ca9ea68fbaf122e7c6e60112581e2972c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T05:59:02.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:02.635+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:03.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:02.811+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:03.138 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:02 smithi028 podman[111053]: 2024-09-21 05:59:02.754467316 +0000 UTC m=+0.016365092 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:59:03.138 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:02 smithi028 podman[111053]: 2024-09-21 05:59:02.860010175 +0000 UTC m=+0.121907942 container create dfbf570d278fe229ffb6bad7de8bb564363016953e1cb29c9c477f979f14a9db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T05:59:03.139 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:03 smithi028 podman[111053]: 2024-09-21 05:59:03.062357712 +0000 UTC m=+0.324255488 container init dfbf570d278fe229ffb6bad7de8bb564363016953e1cb29c9c477f979f14a9db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T05:59:03.139 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:03 smithi028 podman[111053]: 2024-09-21 05:59:03.065738324 +0000 UTC m=+0.327636093 container start dfbf570d278fe229ffb6bad7de8bb564363016953e1cb29c9c477f979f14a9db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T05:59:03.139 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:03 smithi028 podman[111053]: 2024-09-21 05:59:03.137278748 +0000 UTC m=+0.399176523 container attach dfbf570d278fe229ffb6bad7de8bb564363016953e1cb29c9c477f979f14a9db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T05:59:03.267 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:02.793+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:03.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:03.265+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:03.561 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 21 05:59:03 smithi028 podman[111053]: 2024-09-21 05:59:03.262494984 +0000 UTC m=+0.524392759 container died dfbf570d278fe229ffb6bad7de8bb564363016953e1cb29c9c477f979f14a9db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:59:03.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:03.182+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:04.009 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:03.641+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:04.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.3.service' 2024-09-21T05:59:04.076 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:59:04.077 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T05:59:04.077 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T05:59:04.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.4 2024-09-21T05:59:04.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:03.857+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:04.257 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:03.794+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:04.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:04.148+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:04.391 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:04 smithi129 systemd[1]: Stopping Ceph osd.4 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:59:04.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:04.260+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:04.864 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:59:04.389+0000 7f56cc5af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:59:04.864 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:59:04.389+0000 7f56cc5af640 -1 osd.4 83 *** Got signal Terminated *** 2024-09-21T05:59:04.864 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4[45968]: 2024-09-21T05:59:04.389+0000 7f56cc5af640 -1 osd.4 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:59:05.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:04.754+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:05.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:04.658+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:05.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:04.862+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:05.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:05.271+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:05.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:05.118+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:06.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:05.721+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:06.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:05.654+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:06.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:05.832+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:06.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:06.274+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:06.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:06.108+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:07.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:06.743+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:07.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:06.688+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:07.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:06.864+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:07.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:07.272+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:07.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:07.070+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:08.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:07.784+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:08.046 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:07.668+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:08.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:08.043+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:08.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:07.864+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:08.758 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:08.292+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:09.013 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:08.634+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:09.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:08.756+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:09.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:09.011+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:09.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:08.901+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:09.726 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:09.329+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:10.003 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:09.678+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:10.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:09.724+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:10.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:10.000+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:10.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:09.894+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:10.728 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:10.344+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:11.019 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:10.629+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:11.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:10.726+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:11.293 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:11.050+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:11.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:10.903+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:11.761 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:11.322+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:12.023 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:11.658+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:12.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:11.759+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:12.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:11.860+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:12.297 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:12.021+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:12.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:12.275+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:13.008 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:12.647+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:13.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:12.816+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:13.270 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:12.806+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:13.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:13.006+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:13.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:13.268+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:13.965 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:13.636+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:14.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:13.820+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:14.244 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:13.830+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:14.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:13.963+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:14.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:14.242+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:14.958 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:14.636+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:15.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:14.841+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:15.222 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:14.812+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:15.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:14.956+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:15.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:15 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:15.219+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:15.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:15.610+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:16.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:15 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:15.778+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:16.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:15.979+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:16.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:15.883+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:16.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:16.199+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:16.937 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:16.591+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:17.241 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:16.797+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:17.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:16.935+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:17.364 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:16.869+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:17.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:17.239+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:17.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:17.557+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:18.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:17.825+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:18.215 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:17.823+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:18.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:17.924+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:18.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:18.212+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:18.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:18.524+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:19.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:18.790+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:19.182 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:18.792+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:19.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:18.939+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:19.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:19.179+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:19.949 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:19.567+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:20.015 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:19.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:20.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:19.753+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:20.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:19.947+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:20.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:20.163+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:20.916 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:20.590+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:20.999 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:20.723+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:21.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:20.748+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:21.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:20.914+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:21.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:20.997+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:21.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:21.150+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:21.911 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:21.574+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:21.968 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:21.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:22.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:21.758+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:22.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:21.565+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:22.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:21.909+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:22.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:21.909+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:22.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:21.965+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:22.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:22.164+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:22.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:22.528+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:23.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:22.563+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:23.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:22.741+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:23.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:22.921+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:23.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:22.706+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:23.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:22.917+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:23.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:22.917+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:23.317 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:23.144+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:23.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:23.521+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:23.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:23.521+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:23.951 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:23.674+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:24.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:23.599+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:24.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:23.716+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:24.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:23.716+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:24.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:23.948+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:24.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:23.948+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:24.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:23.948+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:24.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:24.154+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:24.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:24.497+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:24.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:24.497+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:24.982 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:24.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:25.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:24.554+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:25.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:24.712+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:25.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:24.712+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:25.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:24.990+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:25.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:24.990+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:25.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:24.979+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:25.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:25.130+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:25.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:25.483+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:25.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:25.483+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:25.985 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:25.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:25.985 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:25.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:26.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:25.589+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:26.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:25.685+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:26.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:25.685+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:26.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:25.985+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:26.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:25.985+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:26.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:25.982+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:26.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:26.159+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:26.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:26.460+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:26.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:26.460+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:27.010 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:26.635+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:27.010 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:26.635+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:27.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:26.562+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:27.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:26.706+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:27.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:26.706+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:27.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:27.023+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:27.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:27.023+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:27.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:27.007+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:27.539 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:27.175+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:27.539 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:27.175+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:27.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:27.537+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:27.793 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:27.752+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:27.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:27.753+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:27.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:27.413+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:27.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:27.413+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:28.021 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:27.618+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:28.021 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:27.618+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:28.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:28.012+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:28.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:28.012+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:28.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:28.018+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:28.527 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:28.186+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:28.528 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:28.186+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:28.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:28.525+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:28.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:28.720+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:28.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:28.720+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:28.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:28.403+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:28.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:28.403+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:29.033 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:28.614+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:29.033 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:28.614+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:29.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:29.022+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:29.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:29.022+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:29.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:29.030+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:29.541 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:29.208+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:29.541 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:29.208+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:29.793 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:29.682+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:29.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:29.682+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:29.794 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:29.539+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:29.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:29.393+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:29.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:29.393+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:29.984 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:29.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:29.984 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:29.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:30.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:30.019+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:30.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:30.019+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:30.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:29.981+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:30.506 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:30.184+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:30.506 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:30.184+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:30.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:30.504+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:30.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:30.676+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:30.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:30.676+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:30.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:30.368+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:30.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:30.368+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:30.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:30.574+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:30.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:30.574+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:31.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:30.972+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:31.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:30.972+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:31.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:30.970+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:31.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:31.168+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:31.544 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:31.168+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:31.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:31.332+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:31.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:31.332+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:31.555+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:31.555+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:32.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:31.543+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:32.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:31.725+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:32.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:31.725+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:32.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:31.986+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:32.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:31.986+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:32.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:31.965+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:32.518 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:32.165+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:32.518 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:32.165+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:32.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:32.515+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:32.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:32.744+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:32.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:32.744+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:32.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:32.360+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:32.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:32.360+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:32.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:32.599+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:32.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:32.599+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:33.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:32.939+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:33.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:32.939+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:33.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:32.924+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:33.508 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:33.161+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:33.508 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:33.161+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:33.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:33.506+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:33.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:33.726+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:33.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:33.726+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:33.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:33.338+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:33.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:33.338+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:33.961 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:33.636+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:33.962 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:33.636+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:34.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:33.947+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:34.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:33.947+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:34.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:33.959+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:34.383 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:34.127+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:34.383 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:34.127+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:34.647 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:34 smithi129 bash[67465]: time="2024-09-21T05:59:34Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4 in 30 seconds, resorting to SIGKILL" 2024-09-21T05:59:34.647 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:34 smithi129 podman[67465]: 2024-09-21 05:59:34.576569183 +0000 UTC m=+30.311721443 container died 8984b4e9355ca2a1b41fc53824f02b4933324fffb60deb53a594ef64d63badbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T05:59:34.743 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:34.484+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:34.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:34.310+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:34.811 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:34.310+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:34.939 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:34.645+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:34.939 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:34.645+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:35.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:34.741+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:35.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:34.741+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:35.272 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:34.984+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:35.272 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:34 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:34.984+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:35.323 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:34.953+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:35.323 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67465]: 2024-09-21 05:59:35.037708379 +0000 UTC m=+30.772860641 container cleanup 8984b4e9355ca2a1b41fc53824f02b4933324fffb60deb53a594ef64d63badbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2) 2024-09-21T05:59:35.324 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 bash[67465]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4 2024-09-21T05:59:35.455 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:35.153+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:35.455 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:35.153+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:35.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:35.270+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:35.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:35.270+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:35.614 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67482]: 2024-09-21 05:59:35.322394953 +0000 UTC m=+0.739262501 container remove 8984b4e9355ca2a1b41fc53824f02b4933324fffb60deb53a594ef64d63badbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T05:59:35.614 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@osd.4.service: Main process exited, code=exited, status=137/n/a 2024-09-21T05:59:35.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:35.453+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:35.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:35.697+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:35.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:35.697+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:35.910 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67567]: 2024-09-21 05:59:35.555068718 +0000 UTC m=+0.016174114 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T05:59:35.910 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67567]: 2024-09-21 05:59:35.656772624 +0000 UTC m=+0.117878004 container create c4d30afc08ad510d59043d03c0697b13719a8b773db58c596e1cb4ff7704111a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T05:59:35.911 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67567]: 2024-09-21 05:59:35.909025976 +0000 UTC m=+0.370131359 container init c4d30afc08ad510d59043d03c0697b13719a8b773db58c596e1cb4ff7704111a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T05:59:35.911 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:35.668+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:35.911 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:35.668+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:36.305 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:35.999+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:36.305 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:35.999+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:36.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:35.958+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:36.364 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67567]: 2024-09-21 05:59:35.913271036 +0000 UTC m=+0.374376419 container start c4d30afc08ad510d59043d03c0697b13719a8b773db58c596e1cb4ff7704111a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T05:59:36.364 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:35 smithi129 podman[67567]: 2024-09-21 05:59:35.984048677 +0000 UTC m=+0.445154067 container attach c4d30afc08ad510d59043d03c0697b13719a8b773db58c596e1cb4ff7704111a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux ) 2024-09-21T05:59:36.364 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 05:59:36 smithi129 podman[67567]: 2024-09-21 05:59:36.103833492 +0000 UTC m=+0.564938873 container died c4d30afc08ad510d59043d03c0697b13719a8b773db58c596e1cb4ff7704111a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T05:59:36.505 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:36.127+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:36.505 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:36.127+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:36.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:36.303+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:36.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:36.303+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:36.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T05:59:36.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T05:59:36.780 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T05:59:36.780 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T05:59:36.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.5 2024-09-21T05:59:36.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:36.503+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:36.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:36.727+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:36.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:36.727+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:36.997 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:36.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:36.997 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:36.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:37.084 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:36 smithi181 systemd[1]: Stopping Ceph osd.5 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T05:59:37.274 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:36.979+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:37.274 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:36 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:36.979+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:37.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:36.995+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:37.536 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:37.081+0000 7f4b27d80640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T05:59:37.536 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:37.081+0000 7f4b27d80640 -1 osd.5 83 *** Got signal Terminated *** 2024-09-21T05:59:37.536 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:37.081+0000 7f4b27d80640 -1 osd.5 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T05:59:37.536 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:37.089+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:37.536 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:37.089+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:37.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:37.272+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:37.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:37.272+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:37.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:37.534+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:37.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:37.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:37.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:37.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:37.960 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:37.640+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:37.960 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:37.640+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:38.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:38.019+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:38.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:38.019+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:38.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:38.229+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:38.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:38.229+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:38.363 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:37.958+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:38.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:38.096+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:38.544 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:38.096+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:38.956 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:38.670+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:38.956 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:38.670+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:39.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:38.574+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:39.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:38.684+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:39.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:38.684+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:39.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:38.990+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:39.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:38.990+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:39.218+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:39.218+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:39.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:38.954+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:39.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:39.094+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:39.544 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:39.094+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:39.984 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:39.702+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:39.984 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:39.702+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:40.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:39.607+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:40.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:39.673+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:40.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:39.673+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:40.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:40.005+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:40.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:40.005+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:40.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:40.252+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:40.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:40.252+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:40.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:39.982+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:40.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:40.090+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:40.544 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:40.090+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:41.014 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:40.716+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:41.014 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:40.716+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:41.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:40.592+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:41.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:40.687+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:41.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:40.687+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:41.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:41.010+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:41.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:41.010+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:41.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:41.236+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:41.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:41.236+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:41.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:41.011+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:41.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:41.068+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:41.543 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:41.068+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:42.004 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:41.679+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:42.004 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:41.679+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:42.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:41.576+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:42.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:41.675+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:42.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:41.675+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:42.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:42.034+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:42.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:42.034+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:42.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:42.230+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:42.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:42.230+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:42.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:42.001+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:42.538 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:42.042+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.758483+0000 front 2024-09-21T05:57:42.758544+0000 (oldest deadline 2024-09-21T05:58:52.758988+0000) 2024-09-21T05:59:42.538 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 05:59:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5[46509]: 2024-09-21T05:59:42.042+0000 7f4b2436b640 -1 osd.5 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:52.759313+0000 front 2024-09-21T05:57:52.759393+0000 (oldest deadline 2024-09-21T05:59:26.262261+0000) 2024-09-21T05:59:42.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:42.536+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:42.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:42.644+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:42.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:42.644+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:43.017 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:42.651+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:43.017 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:42.651+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:43.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:43.004+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:43.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:43.004+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:43.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:43.230+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:43.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:43.230+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:43.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:43.014+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:43.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:43.538+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:43.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:43.600+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:43.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:43.600+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:43.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:43.603+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:43.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:43.603+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:44.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:44.043+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:44.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:44.043+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:44.193+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:44.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:44.193+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:44.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:44.064+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:44.794 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:44.514+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:44.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:44.623+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:44.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:44.623+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:44.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:44.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:44.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:44.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:45.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:45.030+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:45.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:45.030+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:45.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:45.166+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:45.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:45.166+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:45.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:45.067+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:45.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:45.489+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:45.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:45.610+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:45.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:45.610+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:45.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:45.573+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:45.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:45.573+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:46.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:45.982+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:46.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:45.982+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:46.160+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:46.160+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:46.363 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:46.081+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:46.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:46.530+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:46.793 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:46.657+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:46.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:46.657+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:46.882 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:46.614+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:46.882 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:46.614+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:47.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:46.982+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:47.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:46.982+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:47.147+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:47.147+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:47.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:47.060+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:47.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:47.501+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:47.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:47.675+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:47.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:47.675+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:47.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:47.597+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:47.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:47.597+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:48.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:47.979+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:48.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:47.979+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:48 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:48.197+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:48 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:48.197+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:48.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:48.075+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:48.794 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:48.512+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:48.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:48.718+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:48.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:48.718+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:49.080 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:48.619+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:49.080 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:48.619+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:49.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:48 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:48.995+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:49.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:48 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:48.995+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:49.223+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:49.223+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:49.363 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:49.078+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:49.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:49.611+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:49.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:49.611+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:50.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:49.551+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:50.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:49.684+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:50.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:49.684+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:50.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:49.945+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:50.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:49.945+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:50.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:50.194+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:50.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:50.194+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:50.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:50.034+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:50.631 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:50.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:50.631 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:50.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:51.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:50.572+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:51.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:50.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:51.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:50.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:51.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:50.958+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:51.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:50.958+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:51.194+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:51.194+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:51.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:50.994+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:51.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:51.597+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:51.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:51.597+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:52.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:51.593+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:52.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:51.634+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:52.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:51.634+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:52.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:51.944+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:52.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:51.944+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:52.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:52.147+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:52.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:52.147+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:52.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:52.015+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:52.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:52.596+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:52.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:52.596+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:53.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:52.609+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:53.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:52.659+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:53.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:52.659+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:53.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:52.934+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:53.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:52.934+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:53.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:53.153+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:53.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:53.153+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:53.365 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:53.060+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:53.945 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:53.643+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:53.945 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:53.657+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:53.945 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:53.657+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:54.099 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:53.644+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:54.099 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:53.644+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:54.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:53.903+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:54.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:53.903+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:54.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:53 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:53.903+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:54.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:54.127+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:54.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:54.127+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:54.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:54.127+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:54.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:54.098+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:54.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:54.098+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:55.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:54.614+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:54.707+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:54.707+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:54.707+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T05:59:55.106 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:54.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:55.106 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:54.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:55.106 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:54.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T05:59:55.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:54.894+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:55.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:54.894+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:55.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:54 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:54.894+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:55.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:55.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:55.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:55.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:55.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:55.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:55.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:55.104+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:55.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:55.104+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:56.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:55.597+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:56.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:55.733+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:56.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:55.733+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:56.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:55.733+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T05:59:56.093 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:56.093 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:56.093 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T05:59:56.156 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:55.852+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:56.156 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:55.852+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:56.156 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:55 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:55.852+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:56.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:56.091+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:56.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:56.091+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:56.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:56.154+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:56.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:56.154+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:56.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:56.154+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:56.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:56.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:56.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:56.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:56.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:56.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T05:59:57.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:56.644+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:57.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:56.644+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T05:59:57.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:56.727+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:57.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:56.727+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:57.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:56.727+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T05:59:57.166 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:56.880+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:57.166 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:56.880+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:57.166 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:56 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:56.880+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:57.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:57.054+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:57.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:57.054+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:57.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:57.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:57.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:57.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:57.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:57.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:57.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:57.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:57.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:57.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:57.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:57.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T05:59:58.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:57.690+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:58.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:57.690+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T05:59:58.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:57.715+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:58.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:57.715+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:58.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:57.715+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T05:59:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:57.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:57.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:57 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:57.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:58.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:58.151+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:58.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:58.151+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:58.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:58.151+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:58.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:58.062+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:58.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:58.062+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:58.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:58.609+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:58.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:58.609+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:58.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:58.609+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T05:59:59.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:58.677+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T05:59:59.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:58.677+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T05:59:59.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:58.757+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T05:59:59.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:58.757+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T05:59:59.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:58.757+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T05:59:59.186 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:58.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T05:59:59.186 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:58.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T05:59:59.186 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:58 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:58.908+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T05:59:59.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:59.048+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T05:59:59.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 05:59:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T05:59:59.048+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T05:59:59.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:59.184+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T05:59:59.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:59.184+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T05:59:59.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T05:59:59.184+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T05:59:59.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:59.603+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T05:59:59.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:59.603+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T05:59:59.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 05:59:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T05:59:59.603+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:00.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:59.711+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:00.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 05:59:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T05:59:59.711+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:00.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:59.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:00.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:59.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:00.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 05:59:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T05:59:59.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:00.185 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:59.892+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:00.185 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:59.892+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:00.185 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 05:59:59 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T05:59:59.892+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:00.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:00.077+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:00.365 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:00.077+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:00.563 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:00.183+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:00.563 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:00.183+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:00.563 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:00.183+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:00.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:00.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:00.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:00.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:00.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:00.592+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:01.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:00.700+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:01.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:00.700+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:01.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:00.803+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:01.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:00.803+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:01.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:00.803+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:01.235 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:00.929+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:01.235 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:00.929+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:01.235 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:00 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:00.929+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:01.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:01.102+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:01.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:01.102+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:01.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:01.233+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:01.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:01.233+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:01.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:01.233+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:01.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:01.564+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:01.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:01.564+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:01.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:01 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:01.564+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:02.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:01.745+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:02.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:01.745+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:02.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:01.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:02.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:01.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:02.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:01.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:02.241 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:01.889+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:02.241 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:01.889+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:02.241 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:01.889+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:02.241 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:01 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:01.889+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:02.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:02.084+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:02.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:02.084+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:02.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:02.238+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:02.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:02.239+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:02.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:02.239+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:02.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:02.239+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:02.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:02.587+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:02.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:02.587+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:02.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:02 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:02.587+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:03.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:02.734+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:03.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:02.734+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:03.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:02.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:03.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:02.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:03.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:02.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:03.286 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:02.854+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:03.287 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:02.854+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:03.287 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:02.854+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:03.287 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:02 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:02.854+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:03.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:03.065+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:03.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:03.065+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:03.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:03.285+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:03.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:03.285+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:03.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:03.285+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:03.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:03.285+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:03.553+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:03.553+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:03 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:03.553+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:04.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:03.835+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:04.045 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:03.835+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:04.045 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:03.835+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:04.045 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:03.685+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:04.046 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:03.685+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:04.046 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:03.685+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:04.263 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:03.868+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:04.264 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:03.868+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:04.264 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:03.868+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:04.264 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:03 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:03.868+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:04.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:04.036+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:04.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:04.036+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:04.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:04.036+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:04.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:04.262+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:04.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:04.262+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:04.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:04.262+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:04.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:04.262+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:04.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:04.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:04.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:04.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:04.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:04.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:04.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:04 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:04.568+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:05.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:04.703+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:05.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:04.703+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:05.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:04.703+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:05.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:04.844+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:05.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:04.844+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:05.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:04.844+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:05.249 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:04.870+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:05.250 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:04.870+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:05.250 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:04.870+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:05.250 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:04 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:04.870+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:05.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:05.020+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:05.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:05.020+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:05.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:05.020+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:05.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:05.248+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:05.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:05.248+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:05.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:05.248+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:05.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:05.248+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:05.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:05.541+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:05.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:05.541+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:05.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:05.541+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:05.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:05 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:05.541+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:06.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:05.882+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:06.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:05.882+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:06.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:05.882+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:06.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:05.740+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:06.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:05.740+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:06.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:05.740+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:06.237 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:05.839+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:06.237 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:05.839+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:06.237 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:05.839+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:06.237 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:05 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:05.839+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:06.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:06.069+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:06.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:06.069+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:06.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:06.069+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:06.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:06.235+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:06.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:06.235+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:06.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:06.235+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:06.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:06.235+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:06.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:06.551+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:06.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:06.551+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:06.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:06.551+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:06.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:06 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:06.551+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:07.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:06.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:07.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:06.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:07.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:06.917+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:07.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:06.756+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:07.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:06.756+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:07.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:06.756+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:07.210 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:06.883+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:07.210 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:06.883+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:07.210 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:06.883+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:07.210 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:06 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:06.883+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:07.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:07.080+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:07.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:07.080+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:07.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:07.080+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:07.508 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 bash[68421]: time="2024-09-21T06:00:07Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5 in 30 seconds, resorting to SIGKILL" 2024-09-21T06:00:07.508 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 podman[68421]: 2024-09-21 06:00:07.275040151 +0000 UTC m=+30.300854782 container died 269c4aa36acedabb66b95aa9267cae6dcbf256124246347f2454ab2ddd5a43b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T06:00:07.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:07.208+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:07.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:07.208+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:07.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:07.208+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:07.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:07.208+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:07.774 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 podman[68421]: 2024-09-21 06:00:07.596780238 +0000 UTC m=+30.622594876 container cleanup 269c4aa36acedabb66b95aa9267cae6dcbf256124246347f2454ab2ddd5a43b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-21T06:00:07.774 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 bash[68421]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5 2024-09-21T06:00:07.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:07.515+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:07.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:07.515+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:07.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:07.515+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:07.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:07 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:07.515+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:08.043 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 podman[68437]: 2024-09-21 06:00:07.772952989 +0000 UTC m=+0.496599646 container remove 269c4aa36acedabb66b95aa9267cae6dcbf256124246347f2454ab2ddd5a43b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:07 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@osd.5.service: Main process exited, code=exited, status=137/n/a 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:07.781+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:07.781+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:07.781+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:07.897+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:07.897+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:08.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:07.897+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:08.165 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:07.843+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:08.165 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:07.843+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:08.165 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:07.843+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:08.165 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:07 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:07.843+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:08.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:08.104+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:08.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:08.104+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:08.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:08.104+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:08.503 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.118925721 +0000 UTC m=+0.015831863 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:00:08.503 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.224009682 +0000 UTC m=+0.120915823 container create 7dcf8240c2575a05d891b693a0fc41e2b028f12bfdac019afcf4dffa4341c14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T06:00:08.503 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.409969081 +0000 UTC m=+0.306875229 container init 7dcf8240c2575a05d891b693a0fc41e2b028f12bfdac019afcf4dffa4341c14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T06:00:08.503 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.413270133 +0000 UTC m=+0.310176274 container start 7dcf8240c2575a05d891b693a0fc41e2b028f12bfdac019afcf4dffa4341c14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:08.503 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.501689937 +0000 UTC m=+0.398596091 container attach 7dcf8240c2575a05d891b693a0fc41e2b028f12bfdac019afcf4dffa4341c14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5-deactivate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T06:00:08.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:08.163+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:08.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:08.163+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:08.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:08.163+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:08.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:08.163+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:08.793 INFO:journalctl@ceph.osd.5.smithi181.stdout:Sep 21 06:00:08 smithi181 podman[68528]: 2024-09-21 06:00:08.615111636 +0000 UTC m=+0.512017779 container died 7dcf8240c2575a05d891b693a0fc41e2b028f12bfdac019afcf4dffa4341c14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T06:00:08.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:08.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:08.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:08.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:08.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:08.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:08.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:08 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:08.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:08.894+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:08.894+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:08.894+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:08.799+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:08.799+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:09.132 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:08.799+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:09.139 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:08.847+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:09.139 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:08.847+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:09.140 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:08.847+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:09.140 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:08 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:08.847+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:09.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T06:00:09.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:00:09.229 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T06:00:09.229 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T06:00:09.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.6 2024-09-21T06:00:09.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:09.064+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:09.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:09.064+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:09.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:09.064+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:09.553 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 systemd[1]: Stopping Ceph osd.6 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:00:09.553 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:09.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:09.553 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:09.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:09.553 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:09.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:09.553 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:09.137+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.550+0000 7f627f59c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.550+0000 7f627f59c640 -1 osd.6 83 *** Got signal Terminated *** 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.550+0000 7f627f59c640 -1 osd.6 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.807+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.807+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.807+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:09.810 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:09 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:09.807+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:09.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:09.575+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:09.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:09.575+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:09.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:09.575+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:09.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:09 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:09.575+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:10.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:09.779+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:10.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:09.779+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:10.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:09.779+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:10.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:09.941+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:10.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:09.941+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:10.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:09 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:09.941+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:10.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:10.094+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:10.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:10.094+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:10.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:10.094+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:10.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:10.113+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:10.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:10.113+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:10.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:10.113+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:10.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:10.113+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:10.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:10.595+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:10.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:10.595+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:10.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:10.595+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:10.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:10 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:10.595+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:11.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:10.731+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:11.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:10.731+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:11.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:10.731+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:11.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:10.969+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:11.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:10.969+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:11.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:10 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:10.969+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:11.125 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:10.832+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:11.125 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:10.833+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:11.125 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:10.833+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:11.126 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:10 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:10.833+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:11.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:11.089+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:11.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:11.089+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:11.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:11.089+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:11.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:11.124+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:11.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:11.124+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:11.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:11.124+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:11.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:11.124+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:11.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:11.571+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:11.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:11.571+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:11.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:11.571+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:11.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:11 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:11.571+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:11.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:11.686+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:11.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:11.686+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:11.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:11.686+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:12.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:11.805+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:12.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:11.805+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:12.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:11.805+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:12.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:11 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:11.805+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:12.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:11.940+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:12.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:11.940+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:12.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:11 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:11.940+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:12.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:12.102+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:12.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:12.102+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:12.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:12.102+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:12.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:12.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:12.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:12.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:12.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:12.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:12.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:12.164+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:12.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:12.583+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:12.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:12.583+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:12.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:12.583+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:12.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:12 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:12.583+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:12.989 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:12.659+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:12.990 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:12.659+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:12.990 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:12.659+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:13.134 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:12.850+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:13.134 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:12.850+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:13.134 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:12.850+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:13.134 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:12.850+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:13.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:12.987+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:13.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:12.987+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:13.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:12 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:12.987+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:13.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:13.101+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:13.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:13.101+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:13.364 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:13.101+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:13.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:13.132+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:13.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:13.132+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:13.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:13.132+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:13.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:13.132+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:13.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:13.604+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:13.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:13.604+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:13.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:13.604+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:13.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:13 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:13.604+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:13.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:13.693+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:13.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:13.693+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:13.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:13.693+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:14.120 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:13.845+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:44.491849+0000 front 2024-09-21T05:57:44.491624+0000 (oldest deadline 2024-09-21T05:58:49.092136+0000) 2024-09-21T06:00:14.120 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:13.845+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.893747+0000 front 2024-09-21T05:57:51.893732+0000 (oldest deadline 2024-09-21T05:59:21.096949+0000) 2024-09-21T06:00:14.120 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:13.845+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.398107+0000 front 2024-09-21T05:58:23.398166+0000 (oldest deadline 2024-09-21T05:59:53.202344+0000) 2024-09-21T06:00:14.120 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:13 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6[61367]: 2024-09-21T06:00:13.845+0000 7f627bb87640 -1 osd.6 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.604827+0000 front 2024-09-21T05:59:00.604857+0000 (oldest deadline 2024-09-21T06:00:01.704863+0000) 2024-09-21T06:00:14.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:13.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:14.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:13.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:14.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:13 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:13.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:14.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:14.117+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:14.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:14.117+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:14.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:14.117+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:14.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:14 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:14.117+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:14.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:14.134+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:14.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:14.134+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:14.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:14.134+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:14.979 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:14.701+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:14.980 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:14.701+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:14.980 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:14.701+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:15.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:14.624+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:15.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:14.624+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:15.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:14.624+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:15.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:14 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:14.624+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:15.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:14.977+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:15.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:14.977+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:15.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:14 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:14.977+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:15.561 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:15 smithi028 podman[111169]: 2024-09-21 06:00:15.226282075 +0000 UTC m=+5.781839892 container died e7a04689c453fb8761aa556117e5c082b7b954216bd959eee546c5a66d70316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T06:00:15.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:15.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:15.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:15.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:15.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:15.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:15.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:15 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:15.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:15.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:15.175+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:15.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:15.175+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:15.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:15.175+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:16.013 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:15 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:15.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:16.014 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:15 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:15.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:16.014 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:15 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:15.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:16.037 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:15 smithi028 podman[111169]: 2024-09-21 06:00:15.792598718 +0000 UTC m=+6.348156524 container cleanup e7a04689c453fb8761aa556117e5c082b7b954216bd959eee546c5a66d70316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:16.037 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:15 smithi028 bash[111169]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6 2024-09-21T06:00:16.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:15.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:16.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:15.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:16.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:15.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:16.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:15 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:15.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:16.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.011+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:16.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.011+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:16.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.011+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:16.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111184]: 2024-09-21 06:00:16.035728548 +0000 UTC m=+0.808831805 container remove e7a04689c453fb8761aa556117e5c082b7b954216bd959eee546c5a66d70316b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T06:00:16.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:16.061+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:16.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:16.061+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:16.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:16.061+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:16.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:16 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:16.061+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:16.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:16.177+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:16.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:16.177+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:16.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:16.177+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:16.640 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.353208619 +0000 UTC m=+0.112523656 container create ce09ea06aed9609f0a15b7862ffbf7db0736023b4af7e22c18195d984f81986e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T06:00:16.640 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.256875804 +0000 UTC m=+0.016190814 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:00:16.640 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.572354596 +0000 UTC m=+0.331669591 container init ce09ea06aed9609f0a15b7862ffbf7db0736023b4af7e22c18195d984f81986e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T06:00:16.640 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.575831742 +0000 UTC m=+0.335146737 container start ce09ea06aed9609f0a15b7862ffbf7db0736023b4af7e22c18195d984f81986e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T06:00:16.640 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.638806577 +0000 UTC m=+0.398121590 container attach ce09ea06aed9609f0a15b7862ffbf7db0736023b4af7e22c18195d984f81986e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T06:00:16.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:16.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:16.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:16.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:16.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:16.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:17.061 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 21 06:00:16 smithi028 podman[111274]: 2024-09-21 06:00:16.767904662 +0000 UTC m=+0.527219669 container died ce09ea06aed9609f0a15b7862ffbf7db0736023b4af7e22c18195d984f81986e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T06:00:17.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:16.650+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:17.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:16.650+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:17.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:16.650+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:17.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:16 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:16.650+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:17.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:17.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:17.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:16 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:16.967+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:17.469 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:17.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:17.469 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:17.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:17.469 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:17.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:17.469 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:17.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:17.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T06:00:17.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:00:17.546 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T06:00:17.546 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T06:00:17.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.7 2024-09-21T06:00:17.570 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.161+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:17.571 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.161+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:17.571 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.161+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:17.858 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 systemd[1]: Stopping Ceph osd.7 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:00:17.858 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:17.673+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:17.858 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:17.673+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:17.858 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:17.673+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:17.858 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:17.673+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:17.968 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:17.677+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:17.968 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:17.677+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:17.969 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:17.677+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:18.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.854+0000 7f74afb44640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:00:18.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.854+0000 7f74afb44640 -1 osd.7 83 *** Got signal Terminated *** 2024-09-21T06:00:18.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:17 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:17.854+0000 7f74afb44640 -1 osd.7 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:00:18.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:17.966+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:18.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:17.966+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:18.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:17 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:17.966+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:18.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:18.089+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:18.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:18.089+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:18.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:18.089+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:18.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:18 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:18.089+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:18.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:18.200+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:18.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:18.200+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:18.614 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:18.200+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:18.959 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:18.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:18.960 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:18.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:18.960 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:18.669+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:19.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:18.632+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:19.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:18.632+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:19.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:18.632+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:19.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:18 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:18.632+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:19.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:18.957+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:19.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:18.957+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:19.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:18 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:18.957+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:19.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:19.119+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:19.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:19.119+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:19.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:19.119+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:19.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:19 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:19.119+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:19.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:19.156+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:19.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:19.156+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:19.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:19.156+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:19.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:19.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:19.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:19.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:19.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:19.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:19.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:19 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:19.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:19.946 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:19.648+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:19.946 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:19.648+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:19.947 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:19.648+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:20.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:19.944+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:20.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:19.944+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:20.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:19 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:19.944+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:20.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:20.129+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:20.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:20.129+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:20.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:20.129+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:20.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:20 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:20.129+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:20.593 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:20.139+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:20.593 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:20.139+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:20.593 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:20.139+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:20.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:20.639+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:20.793 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:20.639+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:20.794 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:20.639+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:20.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:20.591+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:20.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:20.591+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:20.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:20.591+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:20.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:20 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:20.591+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:21.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:20.926+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:21.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:20.926+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:21.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:20 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:20.926+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:21.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:21.085+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:21.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:21.085+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:21.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:21.085+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:21.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:21 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:21.085+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:21.587 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:21.150+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:21.587 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:21.150+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:21.587 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:21.150+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:21.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:21.585+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:21.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:21.585+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:21.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:21.585+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:21.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:21 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:21.585+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:21.919 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:21.592+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:21.919 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:21.592+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:21.919 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:21.592+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:22.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:21.917+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:22.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:21.917+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:22.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:21 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:21.917+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:22.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:22.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:22.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:22.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:22.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:22.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:22.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:22.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:22.604 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:22.176+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.919481+0000 front 2024-09-21T05:57:54.919519+0000 (oldest deadline 2024-09-21T05:59:20.423063+0000) 2024-09-21T06:00:22.605 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:22.176+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:23.323921+0000 front 2024-09-21T05:58:23.323881+0000 (oldest deadline 2024-09-21T05:59:53.628339+0000) 2024-09-21T06:00:22.605 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7[52169]: 2024-09-21T06:00:22.176+0000 7f74a8127640 -1 osd.7 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.629764+0000 front 2024-09-21T05:59:00.629686+0000 (oldest deadline 2024-09-21T06:00:03.529593+0000) 2024-09-21T06:00:22.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:22.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:22.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:22.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:22.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:22.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:22.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:22 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:22.602+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:22.879 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:22.626+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:22.879 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:22.626+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:22.879 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:22.626+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:23.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:22.877+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:23.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:22.877+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:23.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:22 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:22.877+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:23.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:23.071+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:23.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:23.071+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:23.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:23.071+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:23.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:23 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:23.071+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:23.862 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:23.602+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:23.862 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:23.602+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:23.862 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:23.602+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:23.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:23.557+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:23.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:23.557+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:23.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:23.557+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:23.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:23 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:23.557+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:24.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:23.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:24.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:23.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:24.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:23 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:23.860+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:24.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:24.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:24.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:24.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:24.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:24.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:24.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:24 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:24.084+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:24.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:24.563+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:24.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:24.563+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:24.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:24.563+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:24.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:24 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:24.563+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:24.854+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:24.854+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:24.854+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:24.635+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:24.635+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:25.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:24 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:24.635+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:25.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:25.037+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:25.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:25.037+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:25.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:25.037+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:25.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:25.037+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:25.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:25.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:25.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:25.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:25.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:25.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:25.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:25 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:25.556+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:26.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:25.664+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:26.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:25.664+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:26.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:25.664+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:26.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:25.816+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:26.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:25.816+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:26.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:25 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:25.816+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:26.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:26.065+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:26.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:26.065+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:26.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:26.065+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:26.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:26 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:26.065+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:26.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:26.581+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:26.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:26.581+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:26.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:26.581+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:26.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:26 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:26.581+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:26.646+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:26.646+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:26.646+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:26.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:26.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:27.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:26 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:26.774+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:27.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:27.041+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:27.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:27.041+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:27.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:27.041+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:27.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:27 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:27.041+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:27.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:27.589+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:27.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:27.589+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:27.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:27.589+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:27.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:27 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:27.589+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:27.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:27.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:27.668+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:27.811+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:27.811+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:28.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:27 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:27.811+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:28.264 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:28.012+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:28.264 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:28.012+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:28.264 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:28.012+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:28.264 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:28 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:28.012+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:28.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:28.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:28.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:28.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:28.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:28.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:28.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:28 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:28.577+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:28.682+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:28.682+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:28.682+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:28.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:28.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:29.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:28 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:28.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:29.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.026+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:29.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.026+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:29.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.026+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:29.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.026+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:29.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:29.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:29.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:29.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:29.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:29.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:29.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:29 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:29.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:29.656+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:29.656+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:29.656+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:29.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:29.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:30.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:29 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:29.839+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:30.200 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:29 smithi129 podman[67697]: 2024-09-21 06:00:29.908261531 +0000 UTC m=+12.153364401 container died 5129eea019cd895525757935ea1d11627234e6fc4d378819af8a328942abf5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:30.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.980+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:30.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.980+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:30.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.980+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:30.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:29 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:29.980+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:30.541 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:30 smithi129 podman[67697]: 2024-09-21 06:00:30.288357435 +0000 UTC m=+12.533460304 container cleanup 5129eea019cd895525757935ea1d11627234e6fc4d378819af8a328942abf5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:30.541 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:30 smithi129 bash[67697]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7 2024-09-21T06:00:30.852 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:30 smithi129 podman[67712]: 2024-09-21 06:00:30.539869772 +0000 UTC m=+0.632498153 container remove 5129eea019cd895525757935ea1d11627234e6fc4d378819af8a328942abf5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T06:00:30.852 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:30.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:30.853 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:30.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:30.853 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:30.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:30.853 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:30.622+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:30.877 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:30.611+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:30.877 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:30.611+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:30.877 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:30.611+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:31.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:30 smithi129 podman[67802]: 2024-09-21 06:00:30.75404559 +0000 UTC m=+0.016207008 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:00:31.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:30 smithi129 podman[67802]: 2024-09-21 06:00:30.857473939 +0000 UTC m=+0.119635357 container create 06f8cd16d444300918c72da1f4f4c7a722384f2f31c33623544362f6c91198f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T06:00:31.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:31 smithi129 podman[67802]: 2024-09-21 06:00:31.084961978 +0000 UTC m=+0.347123396 container init 06f8cd16d444300918c72da1f4f4c7a722384f2f31c33623544362f6c91198f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T06:00:31.114 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:31 smithi129 podman[67802]: 2024-09-21 06:00:31.088358352 +0000 UTC m=+0.350519771 container start 06f8cd16d444300918c72da1f4f4c7a722384f2f31c33623544362f6c91198f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T06:00:31.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:30.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:31.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:30.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:31.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:30 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:30.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:31.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:30.952+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:31.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:30.952+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:31.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:30.952+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:31.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:30 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:30.952+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:31.580 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:31 smithi129 podman[67802]: 2024-09-21 06:00:31.184920422 +0000 UTC m=+0.447081863 container attach 06f8cd16d444300918c72da1f4f4c7a722384f2f31c33623544362f6c91198f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:00:31.580 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 06:00:31 smithi129 podman[67802]: 2024-09-21 06:00:31.286356179 +0000 UTC m=+0.548517600 container died 06f8cd16d444300918c72da1f4f4c7a722384f2f31c33623544362f6c91198f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T06:00:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:31.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:31.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:31.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:31.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:31 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:31.647+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:31.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T06:00:31.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:00:31.964 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T06:00:31.965 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-21T06:00:31.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.8 2024-09-21T06:00:31.988 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:31.651+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:31.988 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:31.651+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:31.988 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:31.651+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:31.988 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:31.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:31.988 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:31.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:31.989 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:31 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:31.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:32.267 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 systemd[1]: Stopping Ceph osd.8 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:00:32.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:31.925+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:32.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:31.925+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:32.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:31.925+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:32.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:31 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:31.925+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:32.543 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.265+0000 7fd02dfff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:00:32.544 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.265+0000 7fd02dfff640 -1 osd.8 83 *** Got signal Terminated *** 2024-09-21T06:00:32.544 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.265+0000 7fd02dfff640 -1 osd.8 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:00:33.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.678+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:33.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.678+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:33.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:32.678+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:33.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:32.815+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:33.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:32.815+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:33.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:32 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:32.815+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:33.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:32.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:33.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:32.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:33.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:32.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:33.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:32 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:32.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:33.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:32.935+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:33.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:32.935+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:33.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:32.935+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:33.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:32 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:32.935+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:34.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:33.720+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:34.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:33.720+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:34.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:33.720+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:34.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:33.817+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:34.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:33.817+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:34.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:33 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:33.817+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:34.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:33.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:34.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:33.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:34.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:33.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:34.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:33.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:34.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:33.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:34.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:33.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:34.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:33.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:34.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:33 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:33.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:34.752+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:34.752+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:34.752+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:34.793+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:34.793+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:35.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:34 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:34.793+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:35.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:34.721+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:35.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:34.721+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:35.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:34.721+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:35.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:34 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:34.721+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:35.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.003+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:35.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.003+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:35.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.003+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:35.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.003+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:35.721+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:35.721+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:35.721+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:35.721+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.428999+0000 front 2024-09-21T05:59:08.428931+0000 (oldest deadline 2024-09-21T06:00:35.632619+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:35.769+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:35.769+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:35.769+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:36.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:35 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:35.769+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:36.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:35.698+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:36.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:35.698+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:36.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:35.698+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:36.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:35 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:35.698+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:36.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:36.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:36.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:36.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:36.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:35 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:35.982+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:37.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:36.724+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.217496+0000 front 2024-09-21T05:57:54.217596+0000 (oldest deadline 2024-09-21T05:59:21.521329+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:36.724+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.422085+0000 front 2024-09-21T05:58:27.422189+0000 (oldest deadline 2024-09-21T05:59:55.925242+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:36.724+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:59:00.426856+0000 front 2024-09-21T05:59:00.426814+0000 (oldest deadline 2024-09-21T06:00:03.326778+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8[52961]: 2024-09-21T06:00:36.724+0000 7fd0265e2640 -1 osd.8 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.428999+0000 front 2024-09-21T05:59:08.428931+0000 (oldest deadline 2024-09-21T06:00:35.632619+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:36.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:36.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:36.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:37.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:36 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:36.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:37.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:36.695+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:37.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:36.695+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:37.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:36.695+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:37.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:36 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:36.695+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:37.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:37.024+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:37.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:37.024+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:37.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:37.024+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:37.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:37.024+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:37.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:37 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:37.024+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:38.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:37.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:38.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:37.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:38.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:37.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:38.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:37.694+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:38.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:37.810+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:38.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:37.810+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:38.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:37.810+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:38.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:37 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:37.810+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:38.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:38.072+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:38.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:38.072+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:38.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:38.072+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:38.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:38.072+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:38.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:38 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:38.072+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:39.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:38.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:39.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:38.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:39.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:38.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:39.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:38 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:38.791+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:39.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:38.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:39.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:38.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:39.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:38.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:39.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:38.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:39.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:38 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:38.678+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:39.051+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:39.051+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:39.051+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:39.051+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:39.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:39 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:39.051+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:40.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:39.718+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:40.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:39.718+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:40.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:39.718+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:40.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:39.718+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:40.115 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:39 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:39.718+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:40.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:39.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:40.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:39.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:40.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:39.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:40.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:39 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:39.838+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:40.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:40.080+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:40.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:40.080+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:40.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:40.080+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:40.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:40.080+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:40.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:40 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:40.080+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:41.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:40.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:41.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:40.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:41.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:40.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:41.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:40.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:41.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:40 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:40.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:41.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:40.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:41.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:40.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:41.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:40.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:41.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:40 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:40.827+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:41.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:41.097+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:41.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:41.097+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:41.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:41.097+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:41.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:41.097+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:41.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:41 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:41.097+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:42.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:41.735+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:42.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:41.735+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:42.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:41.735+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:42.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:41.735+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:42.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:41.735+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:42.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:41.797+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:42.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:41.797+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:42.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:41.797+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:42.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:41 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:41.797+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:42.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:42.060+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:42.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:42.060+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:42.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:42.060+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:42.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:42.060+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:42.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:42 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:42.060+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:43.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:42.766+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:43.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:42.766+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:43.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:42.766+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:43.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:42 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:42.766+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:43.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:42.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:43.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:42.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:43.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:42.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:43.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:42.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:43.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:42 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:42.742+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:43.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:43.066+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:43.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:43.066+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:43.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:43.066+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:43.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:43.066+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:43.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:43 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:43.066+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:44.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:43.736+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:44.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:43.736+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:44.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:43.736+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:44.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:43 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:43.736+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:44.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:43.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:44.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:43.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:44.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:43.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:44.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:43.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:44.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:43 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:43.719+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:44.023+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:44.023+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:44.023+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:44.023+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:44.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:44 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:44.023+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:45.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:44.734+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:45.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:44.734+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:45.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:44.734+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:45.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:44 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:44.734+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:45.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:44.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:45.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:44.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:45.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:44.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:45.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:44.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:45.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:44 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:44.675+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:45.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.017+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:45.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.017+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:45.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.017+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:45.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.017+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:45.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.017+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:45.701 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:45 smithi181 podman[68642]: 2024-09-21 06:00:45.332728822 +0000 UTC m=+13.175069248 container died 53df391d8b5e6df3bc36992c731fad6ddfe64c5268061469832d17a4160d29db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:00:46.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:45.698+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:46.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:45.698+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:46.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:45.698+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:46.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:45.698+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:46.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:45 smithi181 podman[68642]: 2024-09-21 06:00:45.931413449 +0000 UTC m=+13.773753875 container cleanup 53df391d8b5e6df3bc36992c731fad6ddfe64c5268061469832d17a4160d29db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:46.044 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:45 smithi181 bash[68642]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8 2024-09-21T06:00:46.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:45.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:46.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:45.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:46.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:45.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:46.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:45.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:46.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:45 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:45.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.976+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.976+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.976+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.976+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:46.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:45 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:45.976+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:46.485 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68657]: 2024-09-21 06:00:46.186444291 +0000 UTC m=+0.849686448 container remove 53df391d8b5e6df3bc36992c731fad6ddfe64c5268061469832d17a4160d29db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T06:00:46.762 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:46.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:46.762 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:46.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:46.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:46 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:46.683+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.484084363 +0000 UTC m=+0.094567805 container create 251e5600a18ba549be0f46e461c1a7395029b0491cb753322dbcd46ac3a84ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.405763323 +0000 UTC m=+0.016246721 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.686397642 +0000 UTC m=+0.296881039 container init 251e5600a18ba549be0f46e461c1a7395029b0491cb753322dbcd46ac3a84ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.689791126 +0000 UTC m=+0.300274525 container start 251e5600a18ba549be0f46e461c1a7395029b0491cb753322dbcd46ac3a84ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T06:00:46.763 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.761224512 +0000 UTC m=+0.371707918 container attach 251e5600a18ba549be0f46e461c1a7395029b0491cb753322dbcd46ac3a84ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:00:47.043 INFO:journalctl@ceph.osd.8.smithi181.stdout:Sep 21 06:00:46 smithi181 podman[68747]: 2024-09-21 06:00:46.881392521 +0000 UTC m=+0.491875927 container died 251e5600a18ba549be0f46e461c1a7395029b0491cb753322dbcd46ac3a84ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T06:00:47.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:46.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:47.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:46.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:47.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:46.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:47.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:46.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:47.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:46 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:46.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:46.971+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:46.971+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:46.971+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:46.971+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:47.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:46 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:46.971+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:47.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.8.service' 2024-09-21T06:00:47.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:00:47.559 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-21T06:00:47.559 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-21T06:00:47.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.9 2024-09-21T06:00:47.735 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:47.701+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:47.735 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:47.701+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:47.735 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:47.701+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:47.735 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:47 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:47.701+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:47.896 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 systemd[1]: Stopping Ceph osd.9 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:00:47.897 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.894+0000 7f18f349e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:00:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:47.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:47.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:47.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:47.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:48.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:47 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:47.680+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.894+0000 7f18f349e640 -1 osd.9 83 *** Got signal Terminated *** 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.894+0000 7f18f349e640 -1 osd.9 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.995+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.995+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.995+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:48.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.995+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:48.312 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:47 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:47.995+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:49.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:48.728+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:49.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:48.728+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:49.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:48.728+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:49.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:48 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:48.728+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:48.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:48.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:48.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:48.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:49.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:48 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:48.709+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:49.032+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:49.032+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:49.032+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:49.032+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:49.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:49 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:49.032+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:50.015 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:49.669+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:50.016 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:49.669+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:50.016 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:49.669+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:50.016 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:49.669+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:50.016 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:49 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:49.669+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:50.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:49.754+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:50.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:49.754+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:50.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:49.754+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:50.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:49.754+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:50.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:50.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:50.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:50.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:50.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:50.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:50.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:50.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:50.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:50 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:50.069+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:51.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:50.677+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:51.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:50.677+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:51.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:50.677+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:51.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:50.677+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:51.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:50 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:50.677+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:51.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:50.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:51.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:50.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:51.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:50.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:51.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:50 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:50.710+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:51.040+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:51.040+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:51.040+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:51.040+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:51.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:51 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:51.040+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:52.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:51.724+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:52.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:51.724+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:52.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:51.724+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:52.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:51 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:51.724+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:51.638+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:51.638+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:51.638+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:51.638+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:52.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:51 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:51.638+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:52.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:52.078+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:45.211502+0000 front 2024-09-21T05:57:45.211707+0000 (oldest deadline 2024-09-21T05:58:48.711619+0000) 2024-09-21T06:00:52.561 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:52.078+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:54.413927+0000 front 2024-09-21T05:57:54.414055+0000 (oldest deadline 2024-09-21T05:59:22.918246+0000) 2024-09-21T06:00:52.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:52.078+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:24.019518+0000 front 2024-09-21T05:58:24.019535+0000 (oldest deadline 2024-09-21T05:59:53.223325+0000) 2024-09-21T06:00:52.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:52.078+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:59.124357+0000 front 2024-09-21T05:58:59.124393+0000 (oldest deadline 2024-09-21T06:00:02.024560+0000) 2024-09-21T06:00:52.562 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:52 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9[68041]: 2024-09-21T06:00:52.078+0000 7f18eba81640 -1 osd.9 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:07.925748+0000 front 2024-09-21T05:59:07.925797+0000 (oldest deadline 2024-09-21T06:00:35.329907+0000) 2024-09-21T06:00:52.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:52.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:52.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:52.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:52.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:52.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:52.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:52.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:52.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:52 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:52.600+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:53.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:52.771+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:53.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:52.771+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:53.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:52.771+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:53.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:52 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:52.771+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:53.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:53.582+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:53.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:53.582+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:53.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:53.582+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:53.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:53.582+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:53.865 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:53 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:53.582+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:53.888 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:53 smithi028 podman[111390]: 2024-09-21 06:00:53.584902361 +0000 UTC m=+5.822262041 container died ea8413345b442daca4ca500c70eacf15f3ffb8940c156a17d27746cc5baed5ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T06:00:54.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:53.735+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:54.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:53.735+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:54.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:53.735+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:54.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:53.735+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:54.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:54 smithi028 podman[111390]: 2024-09-21 06:00:54.069758314 +0000 UTC m=+6.307117979 container cleanup ea8413345b442daca4ca500c70eacf15f3ffb8940c156a17d27746cc5baed5ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T06:00:54.311 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:54 smithi028 bash[111390]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9 2024-09-21T06:00:54.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:54.601+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:54.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:54.601+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:54.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:54.601+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:54.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:54.601+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:54.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:54 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:54.601+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:54.952 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:54 smithi028 podman[111405]: 2024-09-21 06:00:54.581253612 +0000 UTC m=+0.989156821 container remove ea8413345b442daca4ca500c70eacf15f3ffb8940c156a17d27746cc5baed5ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T06:00:55.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:54.780+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:54.780+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:54.780+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:55.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:54 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:54.780+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:55.267 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:54 smithi028 podman[111492]: 2024-09-21 06:00:54.854335814 +0000 UTC m=+0.016507572 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:00:55.268 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:54 smithi028 podman[111492]: 2024-09-21 06:00:54.955818449 +0000 UTC m=+0.117990191 container create 75ca0fea29f346f496e84b513934fd4f4216d066c7f89eaa1d17921611f6299a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:00:55.268 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:55 smithi028 podman[111492]: 2024-09-21 06:00:55.183100865 +0000 UTC m=+0.345272615 container init 75ca0fea29f346f496e84b513934fd4f4216d066c7f89eaa1d17921611f6299a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T06:00:55.268 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:55 smithi028 podman[111492]: 2024-09-21 06:00:55.186647044 +0000 UTC m=+0.348818791 container start 75ca0fea29f346f496e84b513934fd4f4216d066c7f89eaa1d17921611f6299a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T06:00:55.268 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:55 smithi028 podman[111492]: 2024-09-21 06:00:55.266446486 +0000 UTC m=+0.428618245 container attach 75ca0fea29f346f496e84b513934fd4f4216d066c7f89eaa1d17921611f6299a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T06:00:55.546 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 21 06:00:55 smithi028 podman[111492]: 2024-09-21 06:00:55.393649946 +0000 UTC m=+0.555821697 container died 75ca0fea29f346f496e84b513934fd4f4216d066c7f89eaa1d17921611f6299a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:00:55.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.9.service' 2024-09-21T06:00:56.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:00:56.003 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-21T06:00:56.004 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-21T06:00:56.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.10 2024-09-21T06:00:56.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:56.032 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:56.033 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:56.033 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:56.033 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:55 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:55.623+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:56.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:55.823+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:56.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:55.823+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:56.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:55.823+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:56.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:55 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:55.823+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:56.317 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 systemd[1]: Stopping Ceph osd.10 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:00:56.608 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.314+0000 7f68bbfce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:00:56.608 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.314+0000 7f68bbfce640 -1 osd.10 83 *** Got signal Terminated *** 2024-09-21T06:00:56.608 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.314+0000 7f68bbfce640 -1 osd.10 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:00:56.608 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:56.609 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:56.609 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:56.609 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:56.609 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:56 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:56.606+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:57.295 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:56.869+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:57.296 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:56.869+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:57.296 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:56.869+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:57.296 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:56 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:56.869+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:58.116 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:57.655+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:58.116 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:57.655+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:58.116 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:57.655+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:58.116 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:57.655+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:58.116 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:57 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:57.655+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:58.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:57.907+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:58.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:57.907+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:58.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:57.907+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:58.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:57.907+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:00:59.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:58.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:00:59.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:58.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:00:59.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:58.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:00:59.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:58.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:00:59.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:58 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:58.653+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:00:59.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:58.884+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:00:59.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:58.884+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:00:59.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:58.884+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:00:59.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:58 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:58.884+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:00.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:59.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:01:00.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:59.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:01:00.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:59.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:01:00.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:59.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:01:00.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:00:59 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:00:59.642+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:01:00.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:59.870+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:00.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:59.870+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:00.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:59.870+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:00.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:00:59 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:00:59.870+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:01:00.648+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:43.639356+0000 front 2024-09-21T05:57:43.639484+0000 (oldest deadline 2024-09-21T05:58:47.739478+0000) 2024-09-21T06:01:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:01:00.648+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:53.540893+0000 front 2024-09-21T05:57:53.540925+0000 (oldest deadline 2024-09-21T05:59:25.544956+0000) 2024-09-21T06:01:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:01:00.648+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.845550+0000 front 2024-09-21T05:58:27.845579+0000 (oldest deadline 2024-09-21T05:59:53.648160+0000) 2024-09-21T06:01:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:01:00.648+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-21T05:58:58.949734+0000 front 2024-09-21T05:58:58.949755+0000 (oldest deadline 2024-09-21T06:00:03.649226+0000) 2024-09-21T06:01:01.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:00 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10[58553]: 2024-09-21T06:01:00.648+0000 7f68b45b1640 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:05.350186+0000 front 2024-09-21T05:59:05.350160+0000 (oldest deadline 2024-09-21T06:00:38.053593+0000) 2024-09-21T06:01:01.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:00.893+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:01.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:00.893+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:01.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:00.893+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:01.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:00 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:00.893+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:02.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:01.861+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:02.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:01.861+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:02.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:01.861+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:02.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:01 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:01.861+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:02.447 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:02 smithi129 podman[67916]: 2024-09-21 06:01:02.150410603 +0000 UTC m=+5.955307618 container died 26b55f2e278c7c3931afe12072c3c4b7eec0a63044c4310b7cb75bd8a9304907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:02.851 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:02 smithi129 podman[67916]: 2024-09-21 06:01:02.623592931 +0000 UTC m=+6.428489946 container cleanup 26b55f2e278c7c3931afe12072c3c4b7eec0a63044c4310b7cb75bd8a9304907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:02.851 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:02 smithi129 bash[67916]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10 2024-09-21T06:01:03.114 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:02 smithi129 podman[67952]: 2024-09-21 06:01:02.849754749 +0000 UTC m=+0.702739565 container remove 26b55f2e278c7c3931afe12072c3c4b7eec0a63044c4310b7cb75bd8a9304907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T06:01:03.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:02.859+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:03.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:02.859+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:03.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:02.859+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:03.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:02 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:02.859+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:03.446 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.088582345 +0000 UTC m=+0.016176868 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:01:03.446 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.209076592 +0000 UTC m=+0.136671177 container create 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T06:01:03.446 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.444861627 +0000 UTC m=+0.372456143 container init 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T06:01:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.448187164 +0000 UTC m=+0.375781683 container start 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-21T06:01:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.544851251 +0000 UTC m=+0.472445768 container attach 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T06:01:03.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:03 smithi129 podman[68037]: 2024-09-21 06:01:03.635378721 +0000 UTC m=+0.562973251 container died 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:04.209 INFO:journalctl@ceph.osd.10.smithi129.stdout:Sep 21 06:01:04 smithi129 podman[68060]: 2024-09-21 06:01:04.206195336 +0000 UTC m=+0.561748163 container remove 8e7900feca10e350bd26e42d7cdb93b3a294f70b963eff832e5aa651dec32d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T06:01:04.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.10.service' 2024-09-21T06:01:04.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:03.873+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:04.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:03.873+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:04.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:03.873+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:04.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:03 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:03.873+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:04.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:04.306 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-21T06:01:04.306 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-21T06:01:04.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@osd.11 2024-09-21T06:01:04.602 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 systemd[1]: Stopping Ceph osd.11 for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:04.878 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.600+0000 7fc0571da640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:04.878 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.600+0000 7fc0571da640 -1 osd.11 83 *** Got signal Terminated *** 2024-09-21T06:01:04.878 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.600+0000 7fc0571da640 -1 osd.11 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T06:01:05.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.876+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:05.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.876+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:05.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.876+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:05.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:04 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:04.876+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:06.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:05.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:06.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:05.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:06.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:05.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:06.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:05 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:05.916+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:07.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:06.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:07.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:06.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:07.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:06.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:07.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:06 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:06.875+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:08.293 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:07.874+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:08.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:07.874+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:08.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:07.874+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:08.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:07 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:07.874+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:09.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:08.833+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-21T05:57:42.094259+0000 front 2024-09-21T05:57:42.094431+0000 (oldest deadline 2024-09-21T05:58:49.694735+0000) 2024-09-21T06:01:09.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:08.833+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-09-21T05:57:51.995545+0000 front 2024-09-21T05:57:51.995555+0000 (oldest deadline 2024-09-21T05:59:22.998803+0000) 2024-09-21T06:01:09.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:08.833+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-09-21T05:58:27.699649+0000 front 2024-09-21T05:58:27.699611+0000 (oldest deadline 2024-09-21T05:59:54.304124+0000) 2024-09-21T06:01:09.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:08.833+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-09-21T05:59:08.206949+0000 front 2024-09-21T05:59:08.206759+0000 (oldest deadline 2024-09-21T06:00:35.410429+0000) 2024-09-21T06:01:09.294 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:08 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11[59178]: 2024-09-21T06:01:08.833+0000 7fc04f7bd640 -1 osd.11 83 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-09-21T05:59:39.511258+0000 front 2024-09-21T05:59:39.511349+0000 (oldest deadline 2024-09-21T06:01:08.415696+0000) 2024-09-21T06:01:10.734 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:10 smithi181 podman[68876]: 2024-09-21 06:01:10.414335926 +0000 UTC m=+5.921635257 container died ef3c7559dbff326a88aa45c541f50b4bfc6e7659fcdf2e77cc6301649b97d23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T06:01:10.734 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:10 smithi181 podman[68876]: 2024-09-21 06:01:10.733124126 +0000 UTC m=+6.240423447 container cleanup ef3c7559dbff326a88aa45c541f50b4bfc6e7659fcdf2e77cc6301649b97d23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T06:01:11.043 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:10 smithi181 bash[68876]: ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11 2024-09-21T06:01:11.044 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:10 smithi181 podman[68893]: 2024-09-21 06:01:10.942759612 +0000 UTC m=+0.524745103 container remove ef3c7559dbff326a88aa45c541f50b4bfc6e7659fcdf2e77cc6301649b97d23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T06:01:11.539 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.154747951 +0000 UTC m=+0.015913336 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T06:01:11.539 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.282496812 +0000 UTC m=+0.143662191 container create 65aadf99c00b6ad679ecec087d5ebca9de14e6962d250e9333ac3cbe9b2e7322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T06:01:11.539 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.463067416 +0000 UTC m=+0.324232797 container init 65aadf99c00b6ad679ecec087d5ebca9de14e6962d250e9333ac3cbe9b2e7322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T06:01:11.539 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.466400569 +0000 UTC m=+0.327565949 container start 65aadf99c00b6ad679ecec087d5ebca9de14e6962d250e9333ac3cbe9b2e7322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T06:01:11.539 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.537980508 +0000 UTC m=+0.399145893 container attach 65aadf99c00b6ad679ecec087d5ebca9de14e6962d250e9333ac3cbe9b2e7322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:11.794 INFO:journalctl@ceph.osd.11.smithi181.stdout:Sep 21 06:01:11 smithi181 podman[68984]: 2024-09-21 06:01:11.65114928 +0000 UTC m=+0.512314670 container died 65aadf99c00b6ad679ecec087d5ebca9de14e6962d250e9333ac3cbe9b2e7322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T06:01:12.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@osd.11.service' 2024-09-21T06:01:12.283 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:12.284 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-21T06:01:12.284 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-21T06:01:12.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.a 2024-09-21T06:01:12.599 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:12 smithi028 systemd[1]: Stopping Ceph mds.a for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:12.968 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: 2024-09-21T06:01:12.597+0000 7f0bd39fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:12.968 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:12 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a[74323]: 2024-09-21T06:01:12.597+0000 7f0bd39fe640 -1 mds.a *** got signal Terminated *** 2024-09-21T06:01:17.015 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:16 smithi028 podman[111621]: 2024-09-21 06:01:16.609314869 +0000 UTC m=+4.122644131 container died 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T06:01:17.015 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:17 smithi028 podman[111621]: 2024-09-21 06:01:17.014041936 +0000 UTC m=+4.527371197 container cleanup 284974c4152180d730c00244d48022311135b817c964f15f6c7081193c13ad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-21T06:01:17.283 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 21 06:01:17 smithi028 bash[111621]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-a 2024-09-21T06:01:17.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.a.service' 2024-09-21T06:01:17.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:17.498 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-21T06:01:17.499 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-21T06:01:17.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.d 2024-09-21T06:01:17.829 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:17 smithi028 systemd[1]: Stopping Ceph mds.d for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:18.311 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d[73299]: 2024-09-21T06:01:17.826+0000 7f0143e61640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:18.311 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:17 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d[73299]: 2024-09-21T06:01:17.826+0000 7f0143e61640 -1 mds.d *** got signal Terminated *** 2024-09-21T06:01:21.620 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:21 smithi028 podman[111756]: 2024-09-21 06:01:21.368237212 +0000 UTC m=+3.649789539 container died 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T06:01:21.887 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:21 smithi028 podman[111756]: 2024-09-21 06:01:21.67617809 +0000 UTC m=+3.957730410 container cleanup 5fa80b0b69ff09c9c26137eacc1eb3a955ea91595f2baa257cb597216440b90a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T06:01:21.887 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 21 06:01:21 smithi028 bash[111756]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-d 2024-09-21T06:01:22.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.d.service' 2024-09-21T06:01:22.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:22.077 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-21T06:01:22.077 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-21T06:01:22.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.g 2024-09-21T06:01:22.415 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:22 smithi028 systemd[1]: Stopping Ceph mds.g for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:22.811 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-g[74029]: 2024-09-21T06:01:22.413+0000 7f912a7d6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:22.811 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:22 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-g[74029]: 2024-09-21T06:01:22.413+0000 7f912a7d6640 -1 mds.g *** got signal Terminated *** 2024-09-21T06:01:24.924 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:24 smithi028 podman[111877]: 2024-09-21 06:01:24.666630373 +0000 UTC m=+2.373022466 container died 2596bd3e580dd5060fe74bb724e87c477664818f12131b1b665eeea885489cba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-g, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T06:01:25.299 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:25 smithi028 podman[111877]: 2024-09-21 06:01:25.021647334 +0000 UTC m=+2.728039408 container cleanup 2596bd3e580dd5060fe74bb724e87c477664818f12131b1b665eeea885489cba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-g, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD) 2024-09-21T06:01:25.299 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 21 06:01:25 smithi028 bash[111877]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-g 2024-09-21T06:01:25.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.g.service' 2024-09-21T06:01:25.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:25.473 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-21T06:01:25.473 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-21T06:01:25.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.j 2024-09-21T06:01:25.811 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:25 smithi028 systemd[1]: Stopping Ceph mds.j for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:26.311 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j[72913]: 2024-09-21T06:01:25.817+0000 7f1ca356a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:26.311 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:25 smithi028 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j[72913]: 2024-09-21T06:01:25.817+0000 7f1ca356a640 -1 mds.j *** got signal Terminated *** 2024-09-21T06:01:30.086 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:29 smithi028 podman[111996]: 2024-09-21 06:01:29.823835263 +0000 UTC m=+4.130801816 container died 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T06:01:30.345 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 podman[111996]: 2024-09-21 06:01:30.150926341 +0000 UTC m=+4.457892899 container cleanup 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:30.345 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 bash[111996]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j 2024-09-21T06:01:30.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.j.service' 2024-09-21T06:01:30.615 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 podman[112010]: 2024-09-21 06:01:30.34354321 +0000 UTC m=+0.513395443 container remove 61a456e61e81ada5bcea14dc9f7ebbdd702d70bbc1fcab13452440a6a22abb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-j, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T06:01:30.615 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mds.j.service: Deactivated successfully. 2024-09-21T06:01:30.615 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 systemd[1]: Stopped Ceph mds.j for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T06:01:30.615 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 21 06:01:30 smithi028 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mds.j.service: Consumed 1.099s CPU time. 2024-09-21T06:01:30.626 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:30.626 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-21T06:01:30.626 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-21T06:01:30.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.b 2024-09-21T06:01:30.984 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:30 smithi129 systemd[1]: Stopping Ceph mds.b for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:31.248 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: 2024-09-21T06:01:30.982+0000 7f4bad4dc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:31.249 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:30 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b[63436]: 2024-09-21T06:01:30.982+0000 7f4bad4dc640 -1 mds.b *** got signal Terminated *** 2024-09-21T06:01:32.901 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:32 smithi129 podman[68152]: 2024-09-21 06:01:32.632066215 +0000 UTC m=+1.811559819 container died 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T06:01:33.364 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:32 smithi129 podman[68152]: 2024-09-21 06:01:32.998531937 +0000 UTC m=+2.178025540 container cleanup 360c65f6c9a9277e9a77c088d25aa52fd781361b8cd9f2ed7583178d612af9d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T06:01:33.364 INFO:journalctl@ceph.mds.b.smithi129.stdout:Sep 21 06:01:32 smithi129 bash[68152]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-b 2024-09-21T06:01:33.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.b.service' 2024-09-21T06:01:33.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:33.630 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-21T06:01:33.630 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-21T06:01:33.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.e 2024-09-21T06:01:33.963 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:33 smithi129 systemd[1]: Stopping Ceph mds.e for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:34.364 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e[62883]: 2024-09-21T06:01:33.961+0000 7f427f710640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:34.364 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:33 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e[62883]: 2024-09-21T06:01:33.961+0000 7f427f710640 -1 mds.e *** got signal Terminated *** 2024-09-21T06:01:36.912 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:36 smithi129 podman[68287]: 2024-09-21 06:01:36.63470442 +0000 UTC m=+2.777880842 container died 7ddfa32c3fe645d7ba6b9485d5ba9bece1d45b1116afa4a9428209f7428fe5f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T06:01:37.263 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:37 smithi129 podman[68287]: 2024-09-21 06:01:37.002253209 +0000 UTC m=+3.145429623 container cleanup 7ddfa32c3fe645d7ba6b9485d5ba9bece1d45b1116afa4a9428209f7428fe5f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:01:37.263 INFO:journalctl@ceph.mds.e.smithi129.stdout:Sep 21 06:01:37 smithi129 bash[68287]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-e 2024-09-21T06:01:37.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.e.service' 2024-09-21T06:01:37.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:37.466 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-21T06:01:37.466 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-21T06:01:37.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.h 2024-09-21T06:01:37.864 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:37 smithi129 systemd[1]: Stopping Ceph mds.h for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:37.864 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h[63155]: 2024-09-21T06:01:37.772+0000 7f72f77b3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:37.864 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:37 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h[63155]: 2024-09-21T06:01:37.772+0000 7f72f77b3640 -1 mds.h *** got signal Terminated *** 2024-09-21T06:01:40.924 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:40 smithi129 podman[68405]: 2024-09-21 06:01:40.652699006 +0000 UTC m=+2.961647075 container died 70ca59e840c99fb1024b85c166be5e259ba8e7ad31695c0e7228126cd5c478b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T06:01:41.291 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:41 smithi129 podman[68405]: 2024-09-21 06:01:41.022194502 +0000 UTC m=+3.331142568 container cleanup 70ca59e840c99fb1024b85c166be5e259ba8e7ad31695c0e7228126cd5c478b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T06:01:41.291 INFO:journalctl@ceph.mds.h.smithi129.stdout:Sep 21 06:01:41 smithi129 bash[68405]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-h 2024-09-21T06:01:41.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.h.service' 2024-09-21T06:01:41.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:41.456 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-21T06:01:41.456 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-21T06:01:41.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.k 2024-09-21T06:01:41.614 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:41 smithi129 systemd[1]: Stopping Ceph mds.k for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:42.114 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k[62619]: 2024-09-21T06:01:41.801+0000 7f6e247f3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:42.114 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:41 smithi129 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k[62619]: 2024-09-21T06:01:41.801+0000 7f6e247f3640 -1 mds.k *** got signal Terminated *** 2024-09-21T06:01:44.911 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:44 smithi129 podman[68526]: 2024-09-21 06:01:44.650343539 +0000 UTC m=+2.977844288 container died e80cfed98299d96dc1270939b4fe411048fa92d36518714285dfe68a8ca2fe67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T06:01:45.278 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:45 smithi129 podman[68526]: 2024-09-21 06:01:45.009567234 +0000 UTC m=+3.337067980 container cleanup e80cfed98299d96dc1270939b4fe411048fa92d36518714285dfe68a8ca2fe67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T06:01:45.278 INFO:journalctl@ceph.mds.k.smithi129.stdout:Sep 21 06:01:45 smithi129 bash[68526]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-k 2024-09-21T06:01:45.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.k.service' 2024-09-21T06:01:45.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:45.481 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-21T06:01:45.481 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-21T06:01:45.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.c 2024-09-21T06:01:45.794 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:45 smithi181 systemd[1]: Stopping Ceph mds.c for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:46.051 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: 2024-09-21T06:01:45.890+0000 7efe9740b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:46.051 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:45 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c[64273]: 2024-09-21T06:01:45.890+0000 7efe9740b640 -1 mds.c *** got signal Terminated *** 2024-09-21T06:01:48.941 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:48 smithi181 podman[69087]: 2024-09-21 06:01:48.646770278 +0000 UTC m=+2.974925425 container died 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:49.291 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:48 smithi181 podman[69087]: 2024-09-21 06:01:48.940594662 +0000 UTC m=+3.268749817 container cleanup 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T06:01:49.291 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:48 smithi181 bash[69087]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c 2024-09-21T06:01:49.291 INFO:journalctl@ceph.mds.c.smithi181.stdout:Sep 21 06:01:49 smithi181 podman[69111]: 2024-09-21 06:01:49.183377716 +0000 UTC m=+0.532757271 container remove 602d09a33006f75273ee653fdf43f9e2399de24d94892c361079b79b933f44eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T06:01:49.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.c.service' 2024-09-21T06:01:49.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:49.341 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-21T06:01:49.341 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-21T06:01:49.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.f 2024-09-21T06:01:49.637 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:49 smithi181 systemd[1]: Stopping Ceph mds.f for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:50.043 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: 2024-09-21T06:01:49.634+0000 7f64325f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:50.043 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:49 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f[64006]: 2024-09-21T06:01:49.634+0000 7f64325f2640 -1 mds.f *** got signal Terminated *** 2024-09-21T06:01:52.912 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:52 smithi181 podman[69220]: 2024-09-21 06:01:52.656862987 +0000 UTC m=+3.098224518 container died 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:01:53.293 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:52 smithi181 podman[69220]: 2024-09-21 06:01:52.977449948 +0000 UTC m=+3.418811510 container cleanup 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T06:01:53.294 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:52 smithi181 bash[69220]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f 2024-09-21T06:01:53.294 INFO:journalctl@ceph.mds.f.smithi181.stdout:Sep 21 06:01:53 smithi181 podman[69233]: 2024-09-21 06:01:53.128528937 +0000 UTC m=+0.471229613 container remove 22dda338f9274cb890c5af392e8f25ffd224a0d29e261210780618cf0058cb3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-f, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T06:01:53.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.f.service' 2024-09-21T06:01:53.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:53.390 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-21T06:01:53.390 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-21T06:01:53.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.i 2024-09-21T06:01:53.732 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:53 smithi181 systemd[1]: Stopping Ceph mds.i for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:54.043 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: 2024-09-21T06:01:53.730+0000 7f782ba48640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:54.043 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:53 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i[64545]: 2024-09-21T06:01:53.730+0000 7f782ba48640 -1 mds.i *** got signal Terminated *** 2024-09-21T06:01:56.793 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:56 smithi181 podman[69342]: 2024-09-21 06:01:56.65573934 +0000 UTC m=+3.047536281 container died 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T06:01:57.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:57 smithi181 podman[69342]: 2024-09-21 06:01:57.131482821 +0000 UTC m=+3.523279750 container cleanup 4ef35a78ca721c5bf453331bc6ae1b18b980c15766e620750fcc1b77eb0f6bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T06:01:57.294 INFO:journalctl@ceph.mds.i.smithi181.stdout:Sep 21 06:01:57 smithi181 bash[69342]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-i 2024-09-21T06:01:57.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.i.service' 2024-09-21T06:01:57.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:01:57.540 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-21T06:01:57.540 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-21T06:01:57.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-42500114-77db-11ef-baf5-efdc52797490@mds.l 2024-09-21T06:01:57.793 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:01:57 smithi181 systemd[1]: Stopping Ceph mds.l for 42500114-77db-11ef-baf5-efdc52797490... 2024-09-21T06:01:58.293 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:01:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: 2024-09-21T06:01:57.893+0000 7fcea9918640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T06:01:58.293 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:01:57 smithi181 ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l[63736]: 2024-09-21T06:01:57.893+0000 7fcea9918640 -1 mds.l *** got signal Terminated *** 2024-09-21T06:02:00.987 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:00 smithi181 podman[69465]: 2024-09-21 06:02:00.647318937 +0000 UTC m=+2.891537123 container died 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T06:02:01.263 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 podman[69465]: 2024-09-21 06:02:01.084847215 +0000 UTC m=+3.329065398 container cleanup 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T06:02:01.263 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 bash[69465]: ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l 2024-09-21T06:02:01.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42500114-77db-11ef-baf5-efdc52797490@mds.l.service' 2024-09-21T06:02:01.515 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 podman[69479]: 2024-09-21 06:02:01.260907617 +0000 UTC m=+0.605926410 container remove 17bfa00145c3349a0fa031c5b28e3fba69e74ecad73e523db796e35fe4e196c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-42500114-77db-11ef-baf5-efdc52797490-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T06:02:01.515 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mds.l.service: Deactivated successfully. 2024-09-21T06:02:01.515 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 systemd[1]: Stopped Ceph mds.l for 42500114-77db-11ef-baf5-efdc52797490. 2024-09-21T06:02:01.515 INFO:journalctl@ceph.mds.l.smithi181.stdout:Sep 21 06:02:01 smithi181 systemd[1]: ceph-42500114-77db-11ef-baf5-efdc52797490@mds.l.service: Consumed 2.344s CPU time. 2024-09-21T06:02:01.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T06:02:01.526 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-21T06:02:01.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T06:02:01.741 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:04.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T06:02:04.642 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:07.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T06:02:07.320 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:09.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T06:02:09.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T06:02:09.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T06:02:09.786 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T06:02:09.788 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi028/crash 2024-09-21T06:02:09.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash -- . 2024-09-21T06:02:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T06:02:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-09-21T06:02:09.826 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi129/crash 2024-09-21T06:02:09.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash -- . 2024-09-21T06:02:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T06:02:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-09-21T06:02:09.876 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi181/crash 2024-09-21T06:02:09.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash -- . 2024-09-21T06:02:09.925 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/42500114-77db-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T06:02:09.926 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-09-21T06:02:09.927 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T06:02:09.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-21T06:02:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:2024-09-21T05:48:37.136285+0000 mon.a (mon.0) 1237 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T06:02:09.967 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-21T06:02:09.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[SEC\]' /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-21T06:02:10.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]' /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-21T06:02:10.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[WRN\]' /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-21T06:02:10.202 INFO:teuthology.orchestra.run.smithi028.stdout:2024-09-21T05:48:37.136285+0000 mon.a (mon.0) 1237 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2024-09-21T06:02:10.202 INFO:tasks.cephadm:Compressing logs... 2024-09-21T06:02:10.203 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T06:02:10.245 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T06:02:10.248 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T06:02:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T06:02:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T06:02:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.9.log 2024-09-21T06:02:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T06:02:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T06:02:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.9.log: /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.a.log 2024-09-21T06:02:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T06:02:10.271 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log: /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-21T06:02:10.271 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.g.log 2024-09-21T06:02:10.271 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.a.log: 83.8% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log.gzgzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T06:02:10.271 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-21T06:02:10.272 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.g.log: 90.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T06:02:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.10.log 2024-09-21T06:02:10.275 INFO:teuthology.orchestra.run.smithi129.stderr:: No such file or directory 2024-09-21T06:02:10.275 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T06:02:10.276 INFO:teuthology.orchestra.run.smithi181.stderr:: No such file or directory 2024-09-21T06:02:10.276 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T06:02:10.276 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.i.log 2024-09-21T06:02:10.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log: /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.11.log 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log: /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.11.log: gzip 80.6% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T06:02:10.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- 90.3% /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.l.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log: 87.0% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.8.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.z.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T06:02:10.288 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.10.log: /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.b.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.k.log/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log: 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log: 90.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log: 80.6% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T06:02:10.289 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log 2024-09-21T06:02:10.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.8.log: 91.0% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T06:02:10.291 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T06:02:10.291 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log 2024-09-21T06:02:10.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.d.log 2024-09-21T06:02:10.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-21T06:02:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.d.log: 87.0% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T06:02:10.293 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.y.log: 90.9% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T06:02:10.293 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.h.log 2024-09-21T06:02:10.293 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log: 92.5% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T06:02:10.293 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-21T06:02:10.294 INFO:teuthology.orchestra.run.smithi028.stderr: 95.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.d.log.gz 2024-09-21T06:02:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.j.log 2024-09-21T06:02:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T06:02:10.295 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.h.log: 87.0% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T06:02:10.295 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.e.log 2024-09-21T06:02:10.299 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.c.log 2024-09-21T06:02:10.303 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T06:02:10.303 INFO:teuthology.orchestra.run.smithi028.stderr: 95.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.j.log.gz 2024-09-21T06:02:10.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.z.log: 92.6% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.z.log.gz 2024-09-21T06:02:10.307 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.f.log 2024-09-21T06:02:10.310 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T06:02:10.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T06:02:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.e.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T06:02:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 94.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T06:02:10.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.f.log: /var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T06:02:10.403 INFO:teuthology.orchestra.run.smithi028.stderr: 94.4% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.a.log.gz 2024-09-21T06:02:10.410 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.g.log.gz 2024-09-21T06:02:10.447 INFO:teuthology.orchestra.run.smithi129.stderr: 94.4% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.k.log.gz 2024-09-21T06:02:10.450 INFO:teuthology.orchestra.run.smithi181.stderr: 94.4% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.l.log.gz 2024-09-21T06:02:10.578 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T06:02:10.767 INFO:teuthology.orchestra.run.smithi181.stderr: 91.6% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-21T06:02:10.787 INFO:teuthology.orchestra.run.smithi129.stderr: 91.7% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-21T06:02:11.107 INFO:teuthology.orchestra.run.smithi129.stderr: 95.4% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.e.log.gz 2024-09-21T06:02:11.203 INFO:teuthology.orchestra.run.smithi181.stderr: 94.2% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.f.log.gz 2024-09-21T06:02:11.216 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.h.log.gz 2024-09-21T06:02:11.511 INFO:teuthology.orchestra.run.smithi129.stderr: 94.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.b.log.gz 2024-09-21T06:02:11.562 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-21T06:02:11.756 INFO:teuthology.orchestra.run.smithi181.stderr: 94.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.c.log.gz 2024-09-21T06:02:17.720 INFO:teuthology.orchestra.run.smithi181.stderr: 94.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-mds.i.log.gz 2024-09-21T06:02:20.958 INFO:teuthology.orchestra.run.smithi129.stderr: 93.3% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T06:02:22.039 INFO:teuthology.orchestra.run.smithi028.stderr: 93.2% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.9.log.gz 2024-09-21T06:02:22.141 INFO:teuthology.orchestra.run.smithi181.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T06:02:22.441 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.10.log.gz 2024-09-21T06:02:22.571 INFO:teuthology.orchestra.run.smithi181.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T06:02:23.536 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T06:02:24.130 INFO:teuthology.orchestra.run.smithi181.stderr: 93.2% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.8.log.gz 2024-09-21T06:02:25.949 INFO:teuthology.orchestra.run.smithi129.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T06:02:25.977 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T06:02:25.979 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T06:02:25.979 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m15.716s 2024-09-21T06:02:25.979 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m55.496s 2024-09-21T06:02:25.979 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m2.148s 2024-09-21T06:02:27.783 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T06:02:28.770 INFO:teuthology.orchestra.run.smithi181.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.11.log.gz 2024-09-21T06:02:28.772 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T06:02:28.772 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m18.508s 2024-09-21T06:02:28.772 INFO:teuthology.orchestra.run.smithi181.stderr:user 1m4.448s 2024-09-21T06:02:28.772 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m2.271s 2024-09-21T06:02:28.894 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/42500114-77db-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T06:02:28.896 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-21T06:02:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m18.638s 2024-09-21T06:02:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:user 1m0.823s 2024-09-21T06:02:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m2.096s 2024-09-21T06:02:28.897 INFO:tasks.cephadm:Archiving logs... 2024-09-21T06:02:28.897 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi028/log 2024-09-21T06:02:28.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T06:02:37.734 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi129/log 2024-09-21T06:02:37.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T06:02:45.424 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi181/log 2024-09-21T06:02:45.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T06:02:53.715 INFO:tasks.cephadm:Removing cluster... 2024-09-21T06:02:53.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force 2024-09-21T06:02:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:54.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force 2024-09-21T06:02:54.488 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:54.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42500114-77db-11ef-baf5-efdc52797490 --force 2024-09-21T06:02:55.025 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 42500114-77db-11ef-baf5-efdc52797490 2024-09-21T06:02:55.364 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T06:02:55.364 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T06:02:55.384 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T06:02:55.401 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T06:02:55.416 INFO:tasks.cephadm:Teardown complete 2024-09-21T06:02:55.416 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T06:02:55.427 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T06:02:55.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T06:02:55.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T06:02:55.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T06:02:55.549 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T06:02:55.560 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T06:02:55.560 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T06:02:55.563 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T06:02:55.566 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T06:02:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T06:02:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-21T06:02:55.580 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-21T06:02:55.582 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T06:02:55.582 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T06:02:55.582 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 6 377 24 -207us[ -201us] +/- 44ms 2024-09-21T06:02:55.582 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 +531us[ +536us] +/- 51ms 2024-09-21T06:02:55.582 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -338us[ -333us] +/- 74ms 2024-09-21T06:02:55.583 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T06:02:55.583 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T06:02:55.583 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-21T06:02:55.584 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 -199us[ -195us] +/- 44ms 2024-09-21T06:02:55.584 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 +585us[ +585us] +/- 51ms 2024-09-21T06:02:55.584 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 89 -302us[ -298us] +/- 74ms 2024-09-21T06:02:55.584 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T06:02:55.585 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T06:02:55.585 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-21T06:02:55.585 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 88 -231us[ -224us] +/- 43ms 2024-09-21T06:02:55.585 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 +546us[ +546us] +/- 51ms 2024-09-21T06:02:55.585 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -343us[ -343us] +/- 74ms 2024-09-21T06:02:55.586 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T06:02:55.586 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T06:02:55.600 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T06:02:55.600 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T06:02:55.634 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T06:02:55.724 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T06:02:55.790 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T06:02:55.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T06:02:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:55.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T06:02:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:55.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T06:02:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:55.943 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T06:02:55.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T06:02:55.986 INFO:teuthology.task.internal:Duration was 2028.725063 seconds 2024-09-21T06:02:55.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T06:02:56.016 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T06:02:56.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T06:02:56.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T06:02:56.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T06:02:56.058 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T06:02:56.060 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T06:02:56.062 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T06:02:56.402 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T06:02:56.402 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-21T06:02:56.403 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T06:02:56.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T06:02:56.427 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T06:02:56.466 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T06:02:56.466 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T06:02:56.491 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T06:02:56.491 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T06:02:56.493 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T06:02:56.510 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T06:02:56.558 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T06:02:56.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T06:02:56.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T06:02:56.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T06:02:57.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T06:02:57.744 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T06:02:57.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T06:02:57.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T06:02:57.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T06:02:57.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T06:02:57.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:57.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:57.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:57.862 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-21T06:02:57.866 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-21T06:02:57.894 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-21T06:02:58.183 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:58.199 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:58.240 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T06:02:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T06:02:58.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T06:02:58.275 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T06:02:58.276 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi028 2024-09-21T06:02:58.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T06:02:58.338 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi129 2024-09-21T06:02:58.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T06:02:58.378 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913496/remote/smithi181 2024-09-21T06:02:58.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T06:02:58.417 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T06:02:58.417 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T06:02:58.419 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T06:02:58.421 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T06:02:58.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T06:02:58.484 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T06:02:58.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T06:02:58.518 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T06:02:58.518 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T06:02:58.520 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T06:02:58.523 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T06:02:58.533 INFO:teuthology.orchestra.run.smithi028.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 06:02 /home/ubuntu/cephtest 2024-09-21T06:02:58.538 INFO:teuthology.orchestra.run.smithi129.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 06:02 /home/ubuntu/cephtest 2024-09-21T06:02:58.539 INFO:teuthology.orchestra.run.smithi181.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 06:02 /home/ubuntu/cephtest 2024-09-21T06:02:58.541 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T06:02:58.559 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T06:02:58.647 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/ffsb}} duration: 2028.725063085556 failure_reason: '"2024-09-21T05:48:37.136285+0000 mon.a (mon.0) 1237 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-09-21T06:02:58.647 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T06:02:58.735 INFO:teuthology.run:FAIL